Multi-Domain Self-Supervised Learning
Authors: Neha Mukund Kalibhat
Yogesh Balaji
C. Bayan Bruss
Soheil Feizi
Venue: ICLR 2022 Submitted (2022)
Keywords: self-supervised learning
contrastive learning
multi-domain data
unsupervised learning

Abstract: Contrastive self-supervised learning has recently gained significant attention owing to its ability to learn improved feature representations without the use of label information. Current contrastive learning approaches, however, are only effective when trained on a particular dataset, limiting their utility in diverse multi-domain settings. In fact, training these methods on a combination of several domains often degrades the quality of learned representations compared to the models trained on a single domain. In this paper, we propose a Multi-Domain Self-Supervised Learning (MDSSL) approach that can effectively perform representation learning on multiple, diverse datasets. In MDSSL, we propose a three-level hierarchical loss for measuring the agreement between augmented views of a given sample, agreement between samples within a dataset and agreement between samples across datasets. We show that MDSSL when trained on a mixture of CIFAR-10, STL-10, SVHN and CIFAR-100 produces powerful representations, achieving up to a $25\%$ increase in top-1 accuracy on a linear classifier compared to single-domain self-supervised encoders. Moreover, MDSSL encoders can generalize more effectively to unseen datasets compared to both single-domain and multi-domain baselines. MDSSL is also highly efficient in terms of the resource usage as it stores and trains a single model for multiple datasets leading up to $17\%$ reduction in training time. Finally, for multi-domain datasets where domain labels are unknown, we propose a modified approach that alternates between clustering and MDSSL. Thus, for diverse multi-domain datasets (even without domain labels), MDSSL provides an efficient and generalizable self-supervised encoder without sacrificing the quality of representations in individual domains. 

Under review as a conference paper at ICLR 2022
MULTI-DOMAIN SELF-SUPERVISER LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Contrastive self-supervised learning has recently gained significant attention ow- ing to its ability to learn improved feature representations without the use of label information. Current contrastive learning approaches, however, are only effective when trained on particular dataset, limiting their utility in diverse multi-domain settings. In fact, training these methods on a combination of several domains of- ten degrades the quality of learned representations compared to the models trained on a single domain. In this paper, we propose a Multi-Domane Self-Superrvied Learning (MDSSL) approach that can effectively perform representation learning on multiple, diverse datasets. In MDSSL, we propose a three-level hierarchi- cal loss for measuring the agreement between augmented views of given ple, agreement between samples within a dataset and agreement between samples across datasets. We show that MDSSL when trained on mixture of CIFAR-10. STL-10, SVHN and CIFAR-100 produces powerful representations, achieving up 25% increase in top-1 accuracy on a linear classifier compared single- domain self- self-supervised encoders. Moreover, MDSSL encoders can generalize more effectively to unseen datasets compared to both single-domain and multi- domain baselines. MDSSL is also highly efficient in terms of the resource usage as stores and trains a single model for multiple datasets leading up to 17% reduc- tion in training time. Finally, for multi-domain datasets where domain labels are unknown, we propose a modified approach that alternates between clustering and MDSSL. Thus, for diverse multi-domain datasets (even without domain labels), MDSSL provides an efficient and generalizable self-supervised encoder without sacrificing the quality of representations in individual domains.
INTRODUCTION
Self-supervised contrastive training (Chen et al., 2020; He et al. 2020 Misra & van der Maaten 2020 Caron et al. 2020b) has become popular paradigm for unsupervised representation learning as it shows impressive results on linear classification tasks, almost matching the performance of supervised model trained from scratch. However, find that current self-supervised models only effective when trained on single-domain. This can hinder their deployment in large scale real- world settings where data almost always comes from multiple diverse domains. We illustrate this issue in Table I where we show that a popular self-supervised model, SimCLR (Chen et al.) 2020), trained on CIFAR-10 (Krizhevsky We observe that the top-1 accuracy of linear classifier significantly drops unseen datasets. This means that a different self-supervised model needs to be trained for every new dataset, which can add significant computational overheads given that training these models often require large batch sizes and large number of training epochs (Chen et al. 2020 He al., 2020: Wu et al., 2018).
al. a) does not generalize to other domains at the time.
One potential solution for self-supervised learning on multi- datasets is to train the mod- els on the union of all input domains. Unfortunately, this solution performs poorly and fails to obtain a good performance on every individual dataset and does not generalize well to unseen do- mains. To illustrate this, we trained SimCLR on the union of multiple datasets including CIFAR-10 Krizhevsky et al. aa). CIFAR-100 100 (Krizhevsky et al. b). SVHN (Netzer et al. 2011) and STL-10
(Coates et al.). The trained model unfavorable as it significantly decreases the top-1 accuracy in all training datasets compared to the single-domain baselines (see Table T).

---

Under review as a conference paper at ICLR 2022
Mixture Domains
Tiny-Image
Reshal-50 Encodar
DTD
Latent Reprosentations
unknown
Plauuso-Domaln-Laoos
Clustoring after Warmup
Multi Domain Self- Supervised Learning Objective (MDSSL)
Similartt betwoon
Similartty between semplee within a delisee semples acrces dalasees
Simflatty
Recompute regular intervals
Figure 1: Framework of Multi-Domain Self-Supervised Learning: Let us consider our input as mixture of domains, containing datasets from various sources. We introduce MDSSL, three-level hierarchical self-supprviied learning approach to perform representation learning all of these domains the same time. Using a standard ResNet-50 encoder. we learn latent representations that are optimized using the MDSSI. objective. We support MDSSL under two setups with and without domain labels. When domain labels are not available, first cluster representations to identify sseudo-dmmain-labll and then train MDSSL. We use robust clustering approach, while recomputing clusters at regular intervals.
To tackle these issues, we propose Multi-Domail Self-Supervised Learning (MDSSL), tech- nique for obtaining unified embedder that can be trained on multiple domains. MDSSL, we train the model over the union of multiple datasets using three-level hierarchical loss involving:
Embedding similarities of two views of sample: In the first level, we maximize agree- ment (i.e. the cosine similarity between no-oomallzeed vectors) between embeddings two augmented views of given sample.
Embedding similarities of samples from given dataset: In the second level, we mini- mize the pairwise agreements between embeddings of all samples within a datasct.
Embedding similarities of samples from different datasets: In the third level, we mini- mize the pairwise agreement between samples across all training datasets.
The first two levels ensure that the model learns high quality representations for each individual domain. The third level of the MDSSL loss encourages the model to learn distinguishable represen- tations between domains. This approach assumes that domain labels are known during training.
We also extend MDSSL. to more realistic multi-domain setups where domain labels are unknown. In such scenarios, present iterative approach that alternates between clustering and MDSSL at fixed intervals. We use clustering to detect pseudo-domain-labes for each training dataset and use these labels in the MDSSL loss. We also propose robust version of clustering by reducing outlier noise which further improves the performance of MDSSL in an entirely unsupervised setup.
In summary, the goal of MDSSL is to compute improved latent representations of samples from multiple diverse datasets using single self-supervised model (See Figure T]. We summarize our contributions follows:
2

---

Under review as a conference paper at ICLR 2022
We show that current self self-vupe learning techniques such SimCLR, under multi- domain setups, show degraded performance on downstream linear classification tasks and do not generalize well unseen domains.
We propose Multi-Domate Seef--upeeiiied Learning (MDSSL) that uses new loss function for self-supervised learning that supports training over multiple domains at once and pushes the model to learn distinguishable representations across datasets.
We show that MDSSL trained mixture of CIFAR-10) 10, ST1,-10, SVHN and CIFAR-100, shows 25% increase in top-1 accuracy and is more efficient (See Table D.
We also experiment over DTD and Tiny-mmageNet and show that MDSSL generalizes bet- ter to unseen domains of varying diversity compared to both single-domain SimCLR and multi-ommai SimCLR.
We propose an iterative approach combining MDSSL with clustering to train over multi- domain datasets without the use of domain labels.
We further improve our clustering approach by introducing robust clustering that prevents outlier noise from affecting domain labels.
Table 1: Comparing SimCLR and MDSSL single and multi-domain setups
Top-1 Accuracy
Train Dataset
CIFAR-10
STL-10
SVHN
CIFAR-100
Average
Single-Domain Training
CIFAR-I0 STL-10
92.35
56.71
55.97
75.37
70.10
SunCIR
71.05
77.58
63.81
64.62
SVHN
62.83
46.77
92.42
48.27
CIFAR --00
79.58
55.27
61.16
90.29
71.57
Multi Domain Training
SinCLR
CIEAR-10, CIFAR-100,
82.30
61.41
66.65
73.41
70.94
SVHN, STL-10
MDSSSL.
CIFAR-IO, 10, CIFAR-100,
SVHN, STL-10 (A1 = 1, 0.1)
88.45
65.95
75.35
83.05
78.20
RELATED WORK
Supervised classification techniques involve minimizing a loss function (e.g. the cross-entropy loss) match model predictions to true labels. Unsupervised classification methods, the other hand, learn to classify data without the use of training labels, usually with the use of clustering techniques (Bojanowski & Joulin 2017 Dosovitskiy et al. 2014 YM. al. 2020 Bautista al. 2016 Caron
et al. 2018 2019 Huang et al. 2019).
More recently, new unsupervised techniques called self-supervised representation learning have been proposed. A self-supervised model learns by observing every instance of the given data and assigns its own labels, and then performs classification task (Bojanowski & Joulin 2017 Dosovit- skiy et al, 2014] WW et al.) 2018:) Dosovitskiy et al.) 2016). To simplity the complexity of instance- level classificatin,, a memory bank (Wu et al. 2018: He et al., 2020) can used with the help of contrastive learning (Gutmann & Hyvärinen] 2010 Hjelm al. 2019 van den Oord et al. 2019 Grill et al. 2020). Contrastive learning (Arora et al. 2019 Tosh et al. 2021 Bachman et al. 2019 femperatuue--oontrlllee
cross-entropy loss between positive pairs of similar samples and nega- tive pairs of dissimilar samples. Positive pairs are usually considered as multiple transformations (views) (Tian et al., 2020) of given sample using stochastic data augmentation. SimCLR (Chen jet al., (200)) shows that contrastive learning can be done without the use of a memory bank, using the samples within batch, if have large enough batches. SWAV (Caron et al. 2020aa uses
3

---

Under review as a conference paper at ICLR 2022
mixture of contrastive learning and clustering form swapped prediction problem that can learn even with very small batch sizes. Finally, contrastive learning can benefit from training labels, available with a simple modification of contrasting between samples within class and taking sam- ples of other classes as negatives (Khosla et al.) 2020). Each of these approaches show remarkable linear classification accuracy on single-domain setups.
Extending self-supe learning to multiple diverse domains, other than ImageNet (Russakovsky al. 2015), relatively less explored topic (Wallace & Hariharan 2020). When multiple related domains are available during training, a possible approach is to use mutual information to simultane- ously encode common invariant information and domain- speciifc information of each image (Feng et al.] 2019). In our paper, we focus on a general setup where we combine diverse unrelated domains and evaluate individual domain-specific tasks.
MULTI-DOMAIN SELF-SUPERVISED LEARNING WITH DOMAIN LABELS
In this section, we define the Multi-Domain Self-Supervised Learning (MDSSL) paradigm for D training datasets where domain labels are known. We define x E IR' and x;" E IR" as two trans- formed views of the ith sample from the d dataset, base encoder f()) and a two-layer MI.P projection head g(-) to map a given sample into the latent space. We define the latent representations of the two views of the 7th sample from the dt dataset as z f(g(xx)) E R' and 2, d = (x, )) IR' where r' is the size of each latent representation. We represent maini-batches containing 2N samples (2 views per sample) from D datasets as matrix X E R2NDxr whose corresponding latent representation is denoted by Z E R2DDxr'
{1,..., ,.., D}. Similar to SimCLR, we use
We then calculate a similarity matrix E (2NN that contains the exponential cosine similarity scaled by temperature parameter т, between all the latent representations in given batch. The (i element of is:
N Zj
((.)) =- exp
where 2; E IR' and 2j E IR' are the ,th and jth row of Z, respectively.
S represents the cosine similarity between 2a and 2,
Sif represents the cross-dataset cosine
sd.a
similarity between 2 and zif where d, d' E (1, D}. MDSSL aims to solve the following opti- mization problem:
D
N
max
log
St
(2)
ND
2N
2N
X1
log
adpd
(3)
2ND
d=1
1=1
Ø=1
D
2N
D
2N
A2
log Σ
od,d'
(4)
2ND
d'=1/ 1=1
where 0 is the set of model parameters and 0 parameters. This a three-level hierarchical loss. formed views (x: and x in the latent space. minimizes the similarity between every pair of samples (xf and x²)) within a dataset. {4 minimizes the similarity between pairs of samples (xf and x!") across different datasets. When a 1 and Ag 0, this optimization is simplified to SimCLR (Chen et al. 2020). Therefore, we use \₁ = and X2 = 0 as the baseline in all our experiments.
and A2 0 are tunable regularization maximizes the similarity between two trans-
A1 helps us control the extent to which we want minimize the similarity within domain. We empirically observe that relaxing X from the SimCLR baseline (A1 = 0) to a value slightly less than 1, in fact generates better structure in the latent space by clustering samples within a domain relatively closer compared to samples outside a domain (See Appendix Section AA.). X should always be non-negative
always want minimize the agreement between samples of different

---

Under review as a conference paper at ICLR 2022
datasets. We can implement the MDSSL loss efficiently as it only involves calculating the similarity matrix S once and then selecting elements according to each term mask. As the number of datasets increases, the size of S increases, gradually increasing the running time of MDSSL.
3.1 EXPERIMENTAL SETUP
We use ResNet-50 (He al. 2016) the base encoder (f(-) and a 2-layer MLP projection head (g(-)) for all of our experiments. For data augmentation, we use a combination of random crop, random horizontal flip, random color distortion and random Gaussian blur. In all experiments, the latent representations are in a 128-dimensional space and F = 0.1. We optimize our loss using LARS optimizer < You et al. 2017) with a learning rate of and weight decay of 10-6. We train with a batch size of 1024 and train over 48, 000 iterations. We experiment with the following datasets: CIFAR- 10 (Krizhevsky et al., a). CIFAR- 100 (Krizhevsky et al. (b) STL-10 (Coates et al.), SVHN (Netzer et al. 2011). Finy-ImageNee (Le & Yang, 2015) and DTD (Describable Textures) (Cimpoi et al., 2014). We resize all images to 32x32 all our experiments. We use Nvidia GeForce RTX 2080 GPUs. We measure the quality of representations using the linear evaluation protocol (Kolesnikov ct al., 2019, [Bahman et al.] 2019 van den Oord et al., (019) where we train a linear classifier on top of frozen MDSSL representations and compute the top-1 accuracy of each domain-specifer classification task. Since we train over multiple domains, we compute the top-1 accuracy over each domain to evaluate the overall model performance.
Table 2: Resource utilization of SimCLR and MDSSL when trained on CIFAR-10. 10, STL-10, SVHN and CIFAR-100
Resource
Single-Domait SimCLR
MDSSLL
Training Time (hours) Disk Memory (MB) Compute (GPUs)
42.41
34.95 (-17.59%)
968
242 (-75%)
4
2 (-50%)
3.2 MDSSL PERFORMANCE COMPARED SIMCLR BASELINE
In this section, we analyze the performance of MDSSL and compare it to SimCLR trained on single domains (referred as the single-domain SimCLR) and multiple domains (referred as the multi- domain SimCLR). Table I We observe that, single-domain SimCLR models generalize poorly on unseen datasets. For example, SimCLR trained on CIFAR-10 achieves 92.35% top-1 accuracy on CIFAR-10 samples but only 55.97% on SVHN samples.
summarizes our results on CIFAR- 10, STL-10, SVHN and CIFAR- 100.
We also observe that the multi-domain SimCLR model shows degraded performance when eval- uated on each individual training domain. For example, SimCLR trained on the union of samples from CIFAR-10, STL-10, SVHN and CIFAR-100 100 achieves 82.30% top-1 accuracy CIFAR-10, significantly lower than the performance of the single-domain SimCLR model trained on CIFAR- 10. Our method, MDSSL, shows a significant improvement compared to the multi-domain SimCLR and almost matches the baseline accuracy of single-domain SimCLR models on some of the training domains. Among the average top-1 accuracy, we observe up to 25% improvement from the single- domain SimCLR and 10% improvement from the multi-domain SimCLR (See Table SimCLR would require us to train 4 different single-domain models for these datasets and therefore requires more compute, memory and time. MDSSL, being unified model, significantly outperforms Sim- CLR in terms of resource utilization as shown in Table 2 This makes MDSSL an efficient solution in limited resource environments.
3.3 GENERALIZATION TO UNSEEN DATASETS
In this section evaluate the generalization capacity of MDSSL unseen domains. We consider two setups: in the first case, we use CIFAR-10 10, STL-10 and SVHN as training datasets (domains containing less diverse datasets as their number of classes
< 10) and evaluate the model perfor- mance on unseen datasets of CIFAR- 100, DTD and Tiny ImageNet (highly diverse datasets whose number of classes are 10). In the second case, we use CIFAR-100, DTD and Tiny ImageNet as
5

---

Under review as a conference paper at ICLR 2022
Table 3: Generalization of SimCLR and MDSSL unseen domains
Top-1 Accuracy
Train Dataset
CIFAR-10
STL-10
SVIIN
CIFAR-100
DTD
Tiny- ImageNet
Averuge
CIFAR-10
92.35
56.71
55,97
75.57
40.50
19.95
56,80
SimCRR
SVHN
62.83
46.77
92.42
48.27
36.47
13.42
50.03
CIFAR-100
79.58
55.27
61.16
90.29
42.24
21.36
58.31
Ting--mangeNet
81.67
63.20
53.75
82.69
44.03
37.99
60.55
ImageNer (250K)
16
75.43
50.03
21.00
52.38
Dolii-Domoii Training
Average (Truining domains)
Average (Unseen domains)
Sin CL.
CIFAR-10, STL-10,
SVHN
72.10
71:72
MDSSEL
CIFAR-10, STL-10, SVHN
85,50
65.58
88.05
76.48
49.36
24.49
80.37
50.11
(A1 0.9, = 0.1)
Sin
CIFAR-100 DTD, Tiny ImageNet
MOSSE
CIFAR-100, LTTD,
Tiny ImageNNtt A, 0.05)
81,92
62.89
72.35
83.93
54.77
30.18
56.29
72.38
Sin
ImageNet (250)) CIFAR-100,
52.11
$7.98
MDSSE
ImageNet (250K). CIFAR-100, 100, SVHN
81.86
77.01
78.64
80.16
55.30
33.04
79.40
61.80
A.
our training datasets and assess the performances on CIFAR-10, STL-10 and SVHN. We also add results on ImageNet (250K) which contains 1000 classes, each including 250 samples resized to 32x32.
Table 3 summarizes our results. Among the single-domain SimCLR models, we observe that Sim- CLR trained on Tiny-ImageNer generalizes relatively better than other single-domain models since Tiny-ImageNet is comparatively larger and most diverse. However, the drop in top-1 accuracy of unseen domains from the baseline very significant even for single-domain SimCLR trained on ImageNet (42% drop for SVHN). Similarly, ImageNet (250K) also poorly generalizes to unseen domains.
In first multi- domain setup (with training datasets of CIFAR-10, STL-10 and SVHN), ob- serve that although these training datasets are relatively less diverse, MDSSL generalizes remarkably well more diverse datasets like CIFAR-100, DTD and Tmage MDSSL also outperforms the multi-domain SimCLR in all domains (training and unseen). We observe a similar improve- ment when we train MDSSL on CIFAR-100, DTD and Tiny-ImageNet and on ImageNet (250K), CIFAR- 100 and SVHN. MDSSL outperforms both single and multi-domain SimCLR in terms of generalization capacity. These results highlight that MDSSL favorable solution that achieves good accuracy on training domains and generalizes well to unseen domains.
3.4 EFFECT OF NUMBER OF TRAINING DATASETS
In this section, we discuss the behavior of MDSSL as we increase the number of train- ing domains. We train MDSSL on CIFAR-10 and CIFAR-100 (2-domain baseline). We then add STL-10, SVHN, Tmay-Ima and DTD datasets one by one and train MDSSL.

---

Under review as a conference paper at ICLR 2022
SimCLR (CIFAR-10) Baseline
SimCLR (CIFAR-100) Baseline
In Figure we observe that as the number datasets in-
CIFAR-10
creases, the top-1 accuracy also increases and finally beats the single-domain baseline. Therefore, MDSSL benefits from training over large number of datasets.
CIFAR-100
90
Top-1 Accuracy
3.5 HYPERPARAMETER SELECTION
85
The MDSSL loss is controlled by two regularizers A and A2. as shown Section 3 When λ₁ and X2 0, MDSSL
80
boils down to our baseline, SimCLR (Chen et al.) 2020). As
75
we decrease A1 while fixing A2 = 0, we observe that the in- distribution similarity increases (See Appendix Section A..3
70
and eventually, all samples show a mutual similarity of
Number Training Datasots
Consequently, the top-1 accuracy quickly degrades from the baseline as shown the first plot in Figure 3 This behavior can be explained by Term 3 of the MDSSL loss in Section 3 which measures the mutual similarity between all samples within dataset. Therefore, increases in-distribution similarity without significantly af- fecting the top-1 accuracy. We utilize Term of the MDSSL loss by controlling 22 to ensure that domains are more dis- tinguishable the latent space. Figure 3 the second plot shows the top-1 accuracy as we increase 22. The top-1 accu- racy rises steadily at first, and then drops at around 22 = 0.2. This is because after certain threshold, the in-distribution representations become too similar which makes them harder to classify. Therefore, a good balance should be found between λ and A2 such that achieve favourable top-1 accuracy.
Figure 2: Effect of number of training datasets. In this plot, we show that when the number of train- ing datasets increases in MDSSL, the top-1 accuracy increases and eventually beats the SimCLR single- domain baseline, marked by dotted lines. We train MDSSL CIFAR- 10, C00,R-100, STL-10,, SVHN, Tiny-Imageds and DTD.
fix X 0.9 so that it mildly
CIFAR-1G
CIFAR-10
SVHN
80
SVHN
Top-3 Accuracy
Accuracy
60
0.0
0.2
0.4
0.6
0.0
0.1
0.2
0.3
0.4
0.5
Ag
Figure 3: Top-1 accuracy of MDSSL trained on CIFAR-10 and SVHN with varying λ (similar- ity within dataset) and X2 (similarity across datasets). In the first plot, we observe that increasing X1 from -1 (SimCLR Chen et al. (2020) quickly drops the top-1 accuracy since samples within dataset become more and more indistinguishable In the second plot, when \ = -0.9, the top-1 accuracy steadily improves with A2 until a threshold (A2 0.2) and then drops. These plots show that there is a sweet-spot in selecting a and X such that we achieve high top-1 accuracy.
MULTI-DOMAIN SELF-SUPERVISER LEARNING WITHOUT DOMAIN LABELS
Most real-world multi-domain datasets are unlabelled (i.e., domain label information is not avail- able). In this section, develop extension of MDSSL for such setups by identifying pseudo domain labels via a clustering approach in the latent space. As it is common clustering, we assume the number of domains (denoted by M) is known.

---

Under review as a conference paper at ICLR 2022
In the MDSSL loss (especially in Term 4 need domain labels to compute pairwise similarities of samples from two different domains. To achieve this, we first treat the problem as single-domain self-supervised learning and warm up the MDSSL encoder for the first few training iterations using the optimization described in Section 3 with =>1 = 1, X = 0 and = (i.e., SimCLR RR training on one domain). This warm up helps get somewhat distinguishable representations for samples between M domains and the number of iterations to warm up is determined empirically. At the end of the warm up, we cluster the latent representations of the entire multi-domain dataset into M clusters using K-Means clustering (Hartigan & Wong 1979). Using these clusters as pseudo- domain-labels, we continue training the encoder under the MDSSL loss with X < 1. A2 > and D M. As the training progresses, MDSSL improves the latent structure and therefore, we recompute clusters multiple times (determined empirically) as the training progresses to ensure that improved domain labels are used.
In practice, we observe that clustering does not provide 100% accurate domain labels, especially for datasets that are distributionally similar such CIFAR-10 and STL-10. In such cases, propose to use a robust clustering approach coupled with MDSSL to prevent outlier clustering noise from affecting the MDSSL training. Let us consider MDSSL encoder that is warmed up multi- domain dataset containing M domains. We cluster the representations of this dataset into M clusters with centroids C C2, ...... CM- Before assigning ceeudodommiinllablt first determine if they are outliers or not. If so, we ignore these samples in training MDSSL in the next round. We say a latent sample z, not an outlier if it is significantly closer one of the clustering centroids compared to another. Concretely, z; is not an outlier if
each representation,
Cm 112 12:
max
11 m S M, S S M > 1+6
(5)
22 C, 1
where 2 is defined as an outlier threshold. When is high, it means that the given sample is close to its respective centroid. When approaches 0, it indicates that the sample is almost equidistant from at least two centroids and therefore, may not be reliably clustered into one. We ignore such samples in MDSSL. training. When we perform clustering for the first time, start with 1 and each time we repeat clustering, we decay its value exponentially such that it approaches 0 by the end of training to ensure that at the end, all samples contribute to the MDSSL training.
MDSSL with Clustering
MDSSL with Robust Clustering
CIFAR-10
CIFAR-10
SVHN
60
SVHN
40
40
-20
-20
-40
-40
-60
75
-25
75 -100
-50
-25
50
Figure 4: Latent Space of MDSSL with Clustering: We use TSNE to visualize the latent space of MDSSL with clustering on CIFAR-I0 and SVHN. We observe that clustering helps us distinguish between domains and this improves when we apply robust clustering as shown above.
4.1 PERFORMANCE OF MDSSL TRAINED WITHOUT DOMAIN LABELS
Our experimental setup for training MDSSL without domain labels remains the same as the one we explain in Section 3.1 We perform the SimCLR warmup for 480 iterations and update the clus- ters every 2, 400 iterations going forward. In this section, we consider two mixtures for training datasets: (i) CIFAR-10 10 and SVHN (containing visually dissimilar samples), (ii) CIFAR-10, STL.-10
8

---

Under review as a conference paper at ICLR 2022
(containing visually similar samples). In Figure plot the TSNE of CIFAR- 10 and SVHN sam- ple embeddings while using either clustering or robust clustering approaches. We observe that with clustering, achieve reasonable separation between domains although there several outliers. These outliers are significantly reduced while using robust clustering as descrihed in Equation 5 As result, the clusters are quite well defined and easily distinguishable.
In Figure 5] we plot the top-1 accuracy of 5 training setups: single-domain SimCLR, multi-domain SimCLR, MDSSL (with domain labels), MDSSL (with clustering) and MDSSL (with robust cluster- ing). When trained on CIFAR-I0 10 and SVHN, we observe that MDSSL with clustering outperforms SimCLR on both datasets and on CIFAR- 100 which is an unseen domain. We also observe that MDSSL with clustering seems to generalize better to CIFAR-100 1-0 compared to MDSSL. MDSSL with clustering also outperforms SimCLR when trained on CIFAR-10 and STI--10 which are more visually similar. We also observe that applying robust clustering shows an improvement on all do- mains including unseen domains (CIFAR- 100). These results highlight that clustering is a useful approach to identify representations for seen and unseen domains.
and when coupled with MDSSL, it helps learn better
SImCLE Domain) Bascline
DeLla--omain)
MD5SL (with Domain Labels)
MUSSL with Clustering
Train Datasets CFAR-10, SVHN
Train Datasets CIFAR-10, STL-10
.80
Top-3 Ararwy
80
75
75
70
55
55
50
CIFA5-10
SVHN
OFFA-1000
50
GFFA010
STC100
OFAR-100
Figure 5: MDSSL trained with Clustering: We train MDSSL using clustering and robust cluster- ing on CIFAR-10 and SVHN (left) and CIFAR- 10 and STL-10 (right). We observe that, although domain labels are not used in clustering, we are able to improve the performance of MDSSL com- pared SimCLR.
5 DISCUSSION
We propose Multi-Domain Self-Supervised Learning (MDSSL), unifying approach compute self-supervised representations for range of datasets. We support training MDSSL under two se- tups: with domain labels and without domain labels. We show that MDSSL achieves up to a 25% increase in top-1 accuracy with linear evaluation compared to the SimCLR baseline on combi- nation of CIFAR- 10, STL-10,, SVHN and CIFAR-100. We also show that MDSSIL significantly more efficient than SimCLR in terms of resource (time, compute and memory) utilization, gener- alizes better than SimCLR in multi-domain setting, and benefits from an increase in the number of training datasets. In addition,
propose two versions of clustering that can be coupled with MDSSL when training over multiple domains without the use of domain labels. MDSSL achieves good performance general- general-purpose, purpose, enables training on diverse multi- settings, and can obtain meaningful em- beddings achieving state-of-the-an results both seen (training) and unseen benchmark datasets.
under these entirely unsupervised setups. Our unified approach, MDSSL, is

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We share our code the supplementary materials. We also provide several implementation details ensure reproducibility of all our experiments. In Sections 3.1 and A.5 provide detailed explanation of our training setup including the architecture of our encoder, optimizers, learning rate schedule and training hyperparameters. We explain the process of hyperparameter selection in Sections 3.5 and A.3
7 ETHICS STATEMENT
We use only publicly available datasets which involve classification tasks on general objects, vehi- cles, animals, etc. To the best our knowledge, our work does have a negative impact our society any societal group. However, as with all machine learning models, MDSSL. should not be used datasets that are inherently biased or involve harmful tasks that target or affect any particular regional, cultural societal group. Therefore, before running MDSSL, one must select datasets and downstream tasks such that they are safe and not amplify any social biases.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https:
Miguel A Bautista, Artsiom Sanakoyeu, Ekaterina Tikhoncheva, and Bjorn Ommer. Cliqueenn: Deep unsupervised exemplar learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran As- http://..ttttttt//////eeeeeee (5512226688661222224
sociates, Inc., 2016. URL httptt/pp///ppp..cc..ns..stmt./ neurips ccc/paper/2016/1L1 pdf
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learn- ing. volume 70 of Proceedings of Machine Learning Research, pp. 517-526. PMLR, 06-11 Aug 2017. URL httt:::/proceeeiggggg// ml wttpa//pppe/e///////////aaaaaaaa
html
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Mathhij Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), September 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Asso- ciates, Inc., 2020a. URL htttt:::::/ppooeeeding newriierrgg iiriipss.. pss. cc/ ccc/pappr//222//1ee Papeeeec--Paprr--e pdff
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924 Curran Asso- ciates, Inc., 2020b. URL http::t:://ppceeeettett meweeeddiinnnnenuss.p.eee cc cc/ppaper/220//Elle Paper. .dff
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Leurning Reseurch, pp. 1597-1607. PMLR, 13-18 Jul 2020. URL [ /
//proceeding ngs. ml wtw...pppess.eee/ee/ee wtteesssvvl/////e/// chen2/0/che2 html
M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Adam Coates, Honglak Lee, and Andrew Y. Ng. Stanford stl-10 image dataset. URL https: //cs.stanford.ede www.://ddddddaaaatttteeees//eea
Alexey Dosovitskiy, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Z. Ghahramani, M. Welling. C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL

---

Under review as a conference paper at ICLR 2022
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(9):133-1777,, 2016. doi:
William Falcon et al. https://github.com/// htwp://///ooooeiiggiiinnooorrore
Pytorch lightning.
Gitllub.
Note:
3, 2019.
Zeyu Feng, Chang Xu, and Dacheng Tao. Self-supervised representation learning from multi- domain data. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3244- 3254, 2019.
nan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284 Curran Associates, Inc., 2020. URL pttppp/ss//rcccee..ggo....oo// neurips..
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL/http:://rrceediiii www.ann.mm.......mm..........wwe press /v/ /ttp:////nntnnaaaaa.amamm html
J. A. Hartigan and M. A. Wong. A k-means clustering algorithm. JSTOR: Applied Statistics, 28(1): 100-108, 1979.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016. doi: 10.1109/CVRR.2016.990
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9726-9735, 2020. doi: 10. 0,1100/CVR422200020000000099
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. JEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estima- tion and maximization. In International Conference on Learning Representations, 2019. URL httpps://ppenreviier net. forummid=Bk1r3
Karan Grewal, Phil Bachman, Adam
Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning by neigh- bourbood discovery. In Kamalika Chaudhuri and Ruslan Salakhutdinov (cds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2849-2855 PMLR, 09-15 Jun 2019. URL mlr. w...press/vv///ne
9b. html
Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C.S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Magdalene Y.I.. Ting, Jie Zhu, Christina Li, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Yaou Duan, Viet A.N. Huu, Cindy Wen, Edward D. Zhang, Charlotte L. Zhang, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimin Xia, and Kang Zhang. Identifying medical diagnoses and treatable diseases by aaageebased deep learning. Cell, 172(5): 122 131.e9, 2018. ISSN 0092-8674. doi: http:://doi.org1000011... 2018.02.010. URL https: www. 8 wtwp://wwwwwwww...e.e......////e com/ http:///////eeeeeeaeeeeeee////// 6009286741888 0144444831555
12

---

Under review as a conference paper at ICLR 2022
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18661-18673. Curran Asso- neurips cc./paper/2220/F11e
ciates, Inc., 2020. URL httpptt/p//opoceeedii C8 http:///cccccclloaaaaaaaaaaaaaaa Paper.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting sule-supe visual repre- sentation learning. In 2019 IEEE/VV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1920-1929, 2019. doi: 10.1 109/CCPR.2019.00000
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). a. URL http:://p//. www. ca. www.ttooooooooooo wtt....oo...eeeeae/e22222222.... maxxii/cifaa cifarrraar.//// html
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). b. URL
cs. www.:tooooooooooooooooooooo fabb//kriz/cifatt html
Ya Le and X. Yang. Tiny imagenet visual recognition challenge. 2015.
Ishan Misra and I Laurens van der Maaten, Self-supervised learning of pretext-invariant representa- tions. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6706-6716 16, 2020. doi: 10.1099CCVPR2260.2220.00011
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http::/ufldl.sstanffett stwnford..aoero..o edu/
Dousenumberrs/npss201 httrllooouusuummumueseee
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC)), 115(3):211-255 2015. doi: 12600005126------08881
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 6827-6839, Curran Associates, Inc., 2020. URL
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019.
Bram Wallace and Bharath Hariharan. Extending and analyzing self-supervised learning across domains. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Com- puter Vision - ECCV 2020, pp. 717-734, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58574-7.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Asano YM., Rupprecht C., and Vedaldi A. Self-labelling via simultaneous clustering and rep- resentation learning. In International Conference on Learning Representations, 2020. URL pentts::/oppnrreee
net foom:/
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv: Computer Vision and Pattern Recognition, 2017.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 EXPERIMENTS ON NON-OBIECT-FOCUSEE DATASETS
In this section we diseuss the results on 3 datasets that are not object-foeused i.e, EuroSAT (Helber et al., 2019), Chest X-Ray (Pneumonia) (Kermany et al.) 2018) and DTD (Cimpoi et al. 2014). These datasets are understandably not generalizable to unseen domains as shown Table 4 How- under multi-domain setup where we combine all of these domains, MDSSL shows modest improvement compared to multi-domain SimCLR and almost matches the single-domain baselines.
Table 4: Comparing SimCLR and MDSSL on diverse non-object-focused datasets
Top-1 Accuracy
Train Dataset
EuroSAT
ChestXRay
DTD
Average
Vingl-Domain Training
BuroSAT ChestXRay
88.95
93.57
45.75
76,09
85.03
95.29
46.20
75.50
DTD
86.11
93.41
50.43
76.65
Multi-Doman Training
SimCLR
EuroSAT, ChestXRay,
86.02
94.27
46.78
75.69
DTD
MDSSL
EuroSAT, ChestXRay, DTD
87.10
94.43
50.23
77.25
(A1 0.9, A2 = 0.15)
A.2 COMPARING MDSSL WITH SIMCLR PRE-TRAINED ON IMAGENET
In this section, we use a pre-trained SimCLR encoder from Pytorch Lightning Bolts (Falcon et al. 2019) and train linear classifier several unseen datasets. We resize all images to 32x32 during linear classification to maintain consistency with the rest of our experiments. We realize this may unfair comparison since the encoder is pre-trained 224x224 images. Nevertheless, we observe a significant improvement in generalization of MDSSL over SimCLR pre-trained on full-sized ImageNet all datasets (See Table
Table Comparing SimCLR pre- pre-trained on full ImageNet with MDSSL
Top-1 Accuracy
Train Dataset
Tiny- ImageNet
CIFAR-10
STL-10
SVHN
CIFAR-100
DTD
Average
ImageNet
68.21
58.72
49.05
50.11
47.36
20.85
49.05
Multi Domain Training
MDSSL
CIFAR 10, STL-10,
SVHN
87.50
65.58
88.05
76.48
49.36
24.49
65.24
(A1 0.9, A2 = 0.1)
MDSSL
CIFAR-100 100. DTD, Tiny ImageNet (Ai = 0.9, A22 = 0.05)
81.92
62.89
72.35
83.93
54.77
30.18
64.34
14

---

Under review as a conference paper at ICLR 2022
A.3 HYPERPARAMETER SELECTION
In Figure A.I. we plot the similarity matrices of class-averaged samples of MDSSL trained on CIFAR-10 and SVHN by fixing A2 0 and varying Åι. As explained in Section 3 higher Å in- creases the similarity of samples within domain. We observe that when X (SimCLR), the similarity within domain comparable with the similarity across domains, mean- ing that, domains are indistinguishable. As we increase Y₁ we see that the similarity within domains increases and eventually, all samples show mutual similarity
and X =
In Figure A.1 although similarity within domains increases, we still cannot distinguish between domains. To achieve this, we utilize Term of the MDSSL loss by controlling λ2. In Figure A.2 we vary A2 and fix Àι 1 (first row) and λ -0.9 (second row). As X2 increases, the similarity across domains decreases and each domain become clearly distinguishable. When = -0.9, the effect is seen even at lower values of A2. MDSSL learns simultancously increase similarity within domains while decreasing similarity across domains.
1.0
=1
= 0.9
À -
A, 0
0.8
0.5
GIFA-100
CIFAR-10
CIFAR-10
0.4
0.2
0.0
0.0
0,0
CIFAR-10
SWHN
CIFAR-10
SVHN
-0.2
CIFA-110
SVHN
-0.4
1.0
1e-7=100 0.0
1e-7=1
À 0.5
Av= =
A== -01
ト==
10
005
0.5
-1.0
CIFAR-10
0.4
0.0
0.2
-05
0.0
SVHN
SVHN
1.0
GFAR-30
CIFAR-10
CIFAR-10
SVHN
155
Figure A.I: MDSSL trained CIFAR-10 and SVHN with À2 and varying Å These
heatmaps represent the similarity matrices between class-averaged representations of CIFAR-10 and
SVHN. The first matrix in the first row represents our baseline, SimCLR (Chen al. 2020 As À increases, the mutual similarity between samples within domain increases. When x goes over mutual similarity between all training samples effectively reaches
A.4 RUNNING TIME OF MDSSL
The MDSSL optimization, discussed in Section B is solved by iterating over the number of training datasets (D) in each term. Therefore, the number of datasets increases, the running time of MDSSL will increase accordingly. We use large batch size of 1024 which also accounts for increased running time for larger datasets. Figure A.3 we see that as the number of training datasets increases, number of training hours of MDSSL also increases.
A.5 IMPLEMENTATION DETAILS
Table 6 summarizes the entire architecture each component of MDSSL with the filter and output dimensions for input image size 3 32 x 32. Our implementation of MDSSL is Pyrorc.. We
15

---

Under review as a conference paper at ICLR 2022
1.0
1.0
1.00
A1 -1
À1 m1
Av 0.05
0.8
Ao =
0.1
0.75
9.5
CIFAR-10
CIFAR-10
CIFAR-10
9.4
0.25
0.2
0.2
0.00
SVHN
0.0
0.0
SVHN
-0.25
-0.2
OPAR-10
SWMN
OFAN-10
SWN
0.75
-0.4
1.0
2.00
1.00
A
A1
0.8
d
0.1
0.2
0.8
0.50
CIFAR-10
CIFAR 10
CIFAR 10
0.4
0.2
0.00
0.00
SVHN
SVHN
-0.25
SVHN
-0.25
-0.50
CEAM 30
SVmN
CHARCO0
GIFA--00
SWHN
-0.75
Figure A.2: MDSSL trained on CIFAR-10 and SVHN with 0 -1 (first row) and \ = -0.9
(second row) and varying Ag- These heatmaps represent the similarity matrices between class- averaged representations of CIFAR-10 and SVHN. As λ2 increases, the mutual similarity between samples across domains decreases. When A₁ -0.9 MDSSL learns to push samples within domain closer while simultaneously reducing the similarity of samples across domains.
80
Running Time (hours)
60
20
2
4
5
Number Training Datasets
Figure A.3: Running Time of MDSSL. In this plot, we show how the running time increases when the number of training datasets increases in MDSSL. We train MDSSL on the following datasets in the given order: CIFAR-10, SVHN, STL-10, CIFAR-100 DTD, Tiny-ImageNet.
use ResNet-50 (He et al., 2016) as the base encoder for all our experiments. Since we have multiple datasets during training, we prepare DataLooader for each dataset and load batches of size 1024 from each dataset. We refer these as dataset batches. When the number of training datasets
low, concatenate all dataset batches (X) and pass it through the encoder (f(.)) and projection head (g()) to get Z. However, when the number training datasets increases, X becomes too large and may require
memory encode. In this case, first separately encode every dataset batch and then concatenate all dataset embeddings to get Z. This trick helps us efficiently train MDSSL. on 2 GPUs with 4 training datasets and high batch size of 1024.
In Section 3.I we discuss the experimental setup with hyperparameters for MDSSL training. We summarize these parameters in Table 7 We evaluate MDSSL using the linear evaluation protocol
16

---

Under review as a conference paper at ICLR 2022
Table 6: Architecture of MDSSL. projection head and linear classifier
MDSSL Component
Layer
Output Size
Filters
Conv2d
64 X 16 16
x 7, 64, stride 2, padding 3
BatchNorm
64 X 16 X 16
64
RelU
64 x 16 16
MaxPool2d
64 1 x
3 x 3, stride 2, padding
ResNet-50 Encoder
Bottleneck
256 x 8
planes 64, blocks
Bottleneck
512 x 4 x 4
planes 128, blocks 4
Bottleneck
1024 X 2 x 2
planes 256, blocks 6
Bottleneck
2048 1 X 1
planes 512, blocks 3
AdgpoiveAggPool2d
2048 x 1 x
1 x
Linear
2048
2048
Projection Head
RelU
2048
Linear
128
128
Linear Classifier
Linear
10
10
Table Hyperparameter details for MDSSL encoder, projection head and linear classifier
MDSSL Component
Parameter
Value
Latent Dimension
128
Temperature
0.1
Optimizer
LARS
LR Scheduler
Warmup-Annen
Encoder and Projection Head
Learning Rate
Weight Decay
10-6
Batch Size
1024
Number of Training Iterations
48,000
GPU
Nvidia GeForce RTX 2080
Input Dimension
128
Optimizer
SGD
LR Scheduler
Learning Rate
0.1
Linear Classifier
Weight Decay
Batch Size
1024
Number of Training Iterations GPU
30000
Nvidia GeForce RTX 2080
test time, we discard the projection head (o(.) and keep only the ResNet encoder (f(.)). We freeze the encoder and define a trainable linear layer that maps 128-dimensional features from the encoder to class probabilities. This is our linear classifier. We train this classifier over the frozen embeddings from the ResNet encoder for 100 epochs with a batch size 1024. We use the SGD optimizer with an initial learning rate of 0.1. We summarize all of these parameters in Table 7 We optimize the linear classifier using the cross-entropy loss and calculate the top-1 accuracy at the end of training.
Kolesnikov et al. 2019 Bachman et al., 2019 van den Oord al. 2019).
17


Reviews:
[{"review_id": "bdzeCokk71j", "reviewer": "Reviewer_ZUR1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Ups \n\n1.\tSimple solution. The learning objective includes 3 terms, the first 2 terms (minimize the discrepancy between data with different views and maximizing the data pair from the same domain) are derived from SimCLR and the 3rd term is designed to maximizing the discrepancy between pair of data from different domains. \n2.\tGood results. 25% increase in top-1 accuracy when comparing to SimCLR trained on the union of multi-domain data. \n3.\tGreat presentation and through experimental study. The table presented in the paper are well organized and informative to present every details. Clear and concise. \n\nDowns:\n1.\tIt is not very clear for the motivation in Eq (3). Since \\lambda_1 is always negative, does it mean the approach is trying to maximize the discrepancy of a pair of data from the same domain and maximize those between a pair of data from different domain even further? The choice of \\lambda_1 and \\lambda_2 seems very sensitive, could the author provide the corresponding set-up for each experiment and list them in each table? \n2.\tIt is unclear how one domain data is benefits from others. SimCLR performs pool on unobserved domains. This seems that multi-domain data are crossed in the latent space. MDSSL is superior because it introduces a large margin between each domain data. It hints that performance of a particular domain is upper bounded by SimCLR on the single domain. Is it as expected? How will other domains data help?  \n3.\tWithin domain clusters seems important and may need further research. Since MDSSL performs better than SimCLR on the union of multi-domain data, does MDSSL perform superior on single domain with each cluster as a new domain? \n", "summary_of_the_paper": "The paper extends self-supervised contrastive learning from one single domain to multiple domains and achieve better generation for multi-domain performance. \n\nIntuitively, the paper finds SimCLR, an existing self-supervised contrastive learning algorithm performs pool when takes the union of a multi-domain data (e.g., union of CIFAR10, STL-10, SVHN etc) , and is also weak when training on one specific domain and evaluates on the unobserved domain. To bridge the gap, the paper brings in an additional loss into SimCLR to mitigate the gap across different domains - maximizing the discrepancy between data from different domains. \n\nThis change achieves great improvement when training and evaluating via a unified model and better generation – train on one domain data and evaluate on the other domains.\n", "summary_of_the_review": "Pros: Simple approach and good results for building a unified model for multi-domain data.\n\nCons: The formulation and motivation may need further clarification. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4RtpObWU7j", "reviewer": "Reviewer_stxw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is clearly written and well-organized: The motivation, method, experiments, and results are clearly explained. The loss formulation for MDSSL is sensible.  \n2. All experiments show substantial improvement in performance on downstream tasks over single-domain SSL and naive combination of domains. \n3. Ablation studies on hyperparameters and t-SNE plots are helpful in understanding the effect of  design choices on performance. \n\nWeaknesses (Not weaknesses per se, but some questions/suggestions):\n1. No external baselines: As the authors mention in the paper, the Feng et al. approach based on mutual information is a theoretically well-motivated approach for combining multiple domains for learning SSL representations. Comparing MDSSL with this baseline will strengthen the paper. \n2. Additional analysis along domain diversity: While the authors use fairly diverse datasets in terms of size and label types, they have only used natural, object-focused image datasets. Additional experiments with other data domains---medical images, satellite images etc.---will strengthen the paper. \n3. The paper is missing an experiment with full ImageNet pretraining (which is common in SSL papers), which could clarify if MDSSL has advantage over learning representations using a single, but large and diverse dataset. \n", "summary_of_the_paper": "This paper proposes a method for multi-domain self-supervised representation learning (MDSSL).  The contrastive MDSSL training objective consists of three  terms: (1) a loss  term  that  maximizes the similarity between two transformed views of a sample in the latent space (identical to SimCLR), (2) a loss term that maximizes the similarity between every pair of samples within a dataset,  and  (3)  a loss term  that minimizes the similarity between pairs of samples  across different datasets. Loss terms (2) and (3) are weighted using hyperparameters. Authors perform experiments using common datasets including CIFAR-10, CIFAR-100,  STL-10, TinyImageNet and  SVHN. Experiments show that MDSSL outperforms (1) representations learnt by naively combining all datasets and (2) representations learnt on individual domains,  on average classification accuracy on training domains as well as held-out domains. Authors also perform  ablation studies to explore the effect of loss term hyperparameters on  downstream performance. The paper further extends the basic MDSSL method by adding an assumption that dataset labels are not available, and determine dataset labels using k-means clustering during the training process, and also explore a robust k-means clustering method. Experiments using 2 training domains show that the downstream classification performance of this unsupervised MDSSL method is comparable to original labeled MDSSL. ", "summary_of_the_review": "The paper proposes a simple and sensible method for  multi-domain self-supervised learning that obtains good performance on for both in-domain and out-of-domain problems. Comparison with baselines and experiments with full ImageNet datasets is missing. Overall, I think the paper is marginally above the acceptance threshold. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3ndeeMjN3WX", "reviewer": "Reviewer_KFWz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n\nThe method is clear and clean, and the paper is well organized and laid out. The results also show some nice improvements when working with multiple datasets. Overall, it is a nicely written paper with a clear contribution. \n\nWeaknesses:\n\nThere are some shortcomings in the paper, and some spots where the language should be revised.\n\nFirst, there is a confound of dataset size that the authors should address. The authors focus on very small datasets (SVHN, CIFAR10, and similar), then show that SimCLR does poorly to generalize when you mix them all together. What about if working with very large diverse image datasets, like the full size ImageNet? This should at least be mentioned, since these SSL techniques are typically intended for ImageNet and larger training.\n\nSecond, there is not a good theoretical justification for expression (3). The heart of the method is the contrastive learning objective of the expressions (2-4).  (2) and (3) give you the standard SimCLR objective, which puts same-images close together in the latent space, and different-images farther apart in the latent space. \n\nBut there is a small tweak done here. The authors essentially take expression (2), and break it into 2 cases, the case of 2 different images within the database (they now make the representations CLOSER instead of farther like simCLR), and the case of 2 different databases (expression 4), where they make the representations farther apart (that makes sense). I do not have a theoretical appreciation for the tweak of expression (3), to make different images within a database have similar representations, and why that is what is making the representations generalize better. Is it doing some kind of regularization? Can the authors provide some more justification for this change, and perhaps give a bit more of the SimCLR context as I have described here?\n\n\nminor:\n\n\n- related work, 2nd paragraph, first sentence does a bad job of explaining the goal of SSL. Rephrase to say something like “SSL learns representations that are blah blah. It achieves this by … (then the thing you said). Right now, it doesn’t do SSL justice.\n", "summary_of_the_paper": "\nThe paper presents a contrastive learning approach to self-supervised learning (SSL) that works for multiple domains. \n\nThey show that their approach generalizes better across domains than the baseline algorithm (SimCLR). Their method is basically an extension to SimCLR, which is nice because then SimCLR acts as the perfect baseline model.\n\nThey also provide an approach to using multiple domains when the domain labels are not available (using clustering).\n", "summary_of_the_review": "\nThe paper is clean and shows some improvements, but there is not a clear theoretical argument. Furthermore, the paper is limited by using tiny datasets which limit how well we would expect these results to generalize in practice. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a multi-domain self-supervised representation learning method. Its objective consists of three terms: the first two terms are identical to SimCLR and the last one is to minimize the similarity of pairs across different datasets which is similar to the second term of SimCLR. In the experiment, it tests the methods across multiple common datasets. The method is simple but results are pretty good at the multi-domain setting. It seems to demonstrate the importance of domain clustering and moving the domains apart. However, there are several important questions the paper may need more clarification on: 
1. What is the definition of the domain? How to determine the pair of data is from different domains? What is the motivation/theory that you used to choose those datasets as different domains in your experiment? 
2. Is there any of the public datasets that would cover multiple domains? 
Without solving these questions, I think it would constrain the future research/adoption of the method.
Decision: Reject