Year: 2021

Abstract:
Contrastive self-supervised learning has recently gained significant attention owing to its ability to learn improved feature representations without the use of label information. Current contrastive learning approaches, however, are only effective when trained on a particular dataset, limiting their utility in diverse multi-domain settings. In fact, training these methods on a combination of several domains often degrades the quality of learned representations compared to the models trained on a single domain. In this paper, we propose a Multi-Domain Self-Supervised Learning (MDSSL) approach that can effectively perform representation learning on multiple, diverse datasets. In MDSSL, we propose a three-level hierarchical loss for measuring the agreement between augmented views of a given sample, agreement between samples within a dataset and agreement between samples across datasets. We show that MDSSL when trained on a mixture of CIFAR-10, STL-10, SVHN and CIFAR-100 produces powerful representations, achieving up to a 25% increase in top-1 accuracy on a linear classifier compared to singledomain self-supervised encoders. Moreover, MDSSL encoders can generalize more effectively to unseen datasets compared to both single-domain and multidomain baselines. MDSSL is also highly efficient in terms of the resource usage as it stores and trains a single model for multiple datasets leading up to 17% reduction in training time. Finally, for multi-domain datasets where domain labels are unknown, we propose a modified approach that alternates between clustering and MDSSL. Thus, for diverse multi-domain datasets (even without domain labels), MDSSL provides an efficient and generalizable self-supervised encoder without sacrificing the quality of representations in individual domains.

text: Contrastive self-supervised learning has recently gained significant attention owing to its ability to learn improved feature representations without the use of label information. Current contrastive learning approaches, however, are only effective when trained on a particular dataset, limiting their utility in diverse multi-domain settings. In fact, training these methods on a combination of several domains often degrades the quality of learned representations compared to the models trained on a single domain. In this paper, we propose a Multi-Domain Self-Supervised Learning (MDSSL) approach that can effectively perform representation learning on multiple, diverse datasets. In MDSSL, we propose a three-level hierarchical loss for measuring the agreement between augmented views of a given sample, agreement between samples within a dataset and agreement between samples across datasets. We show that MDSSL when trained on a mixture of CIFAR-10, STL-10, SVHN and CIFAR-100 produces powerful representations, achieving up to a 25% increase in top-1 accuracy on a linear classifier compared to singledomain self-supervised encoders. Moreover, MDSSL encoders can generalize more effectively to unseen datasets compared to both single-domain and multidomain baselines. MDSSL is also highly efficient in terms of the resource usage as it stores and trains a single model for multiple datasets leading up to 17% reduction in training time. Finally, for multi-domain datasets where domain labels are unknown, we propose a modified approach that alternates between clustering and MDSSL. Thus, for diverse multi-domain datasets (even without domain labels), MDSSL provides an efficient and generalizable self-supervised encoder without sacrificing the quality of representations in individual domains.

heading: 1 INTRODUCTION
text: Self-supervised contrastive training (Chen et al., 2020; He et al., 2020; Misra & van der Maaten, 2020; Caron et al., 2020b) has become a popular paradigm for unsupervised representation learning as it shows impressive results on linear classification tasks, almost matching the performance of a supervised model trained from scratch. However, we find that current self-supervised models are only effective when trained on a single-domain. This can hinder their deployment in large scale realworld settings where data almost always comes from multiple diverse domains. We illustrate this issue in Table 1, where we show that a popular self-supervised model, SimCLR (Chen et al., 2020), trained on CIFAR-10 (Krizhevsky et al., a) does not generalize to other domains at the test time. We observe that the top-1 accuracy of a linear classifier significantly drops on unseen datasets. This means that a different self-supervised model needs to be trained for every new dataset, which can add significant computational overheads given that training these models often require large batch sizes and a large number of training epochs (Chen et al., 2020; He et al., 2020; Wu et al., 2018).
One potential solution for self-supervised learning on multi-domain datasets is to train the models on the union of all input domains. Unfortunately, this solution performs poorly and fails to obtain a good performance on every individual dataset and does not generalize well to unseen domains. To illustrate this, we trained SimCLR on the union of multiple datasets including CIFAR-10 (Krizhevsky et al., a), CIFAR-100 (Krizhevsky et al., b), SVHN (Netzer et al., 2011) and STL-10 (Coates et al.). The trained model is unfavorable as it significantly decreases the top-1 accuracy in all training datasets compared to the single-domain baselines (see Table 1).
To tackle these issues, we propose Multi-Domain Self-Supervised Learning (MDSSL), a technique for obtaining a unified embedder that can be trained on multiple domains. In MDSSL, we train the model over the union of multiple datasets using a three-level hierarchical loss involving:
• Embedding similarities of two views of a sample: In the first level, we maximize agreement (i.e. the cosine similarity between l2-normalized vectors) between embeddings of two augmented views of a given sample.
• Embedding similarities of samples from a given dataset: In the second level, we minimize the pairwise agreements between embeddings of all samples within a dataset.
• Embedding similarities of samples from different datasets: In the third level, we minimize the pairwise agreement between samples across all training datasets.
The first two levels ensure that the model learns high quality representations for each individual domain. The third level of the MDSSL loss encourages the model to learn distinguishable representations between domains. This approach assumes that domain labels are known during training.
We also extend MDSSL to more realistic multi-domain setups where domain labels are unknown. In such scenarios, we present an iterative approach that alternates between clustering and MDSSL at fixed intervals. We use clustering to detect pseudo-domain-labels for each training dataset and use these labels in the MDSSL loss. We also propose a robust version of clustering by reducing outlier noise which further improves the performance of MDSSL in an entirely unsupervised setup.
In summary, the goal of MDSSL is to compute improved latent representations of samples from multiple diverse datasets using a single self-supervised model (See Figure 1). We summarize our contributions as follows:
Train Dataset Top-1 AccuracyCIFAR-10 STL-10 SVHN CIFAR-100 Average
Single-Domain Training
Si m
C LR CIFAR-10 92.35 56.71 55.97 75.37 70.10 STL-10 71.05 77.58 46.06 63.81 64.62 SVHN 62.83 46.77 92.42 48.27 62.57 CIFAR-100 79.58 55.27 61.16 90.29 71.57
Multi-Domain Training
Si m
C LR CIFAR-10, CIFAR-100, 82.30 61.41 66.65 73.41 70.94SVHN, STL-10
M D
SS L CIFAR-10, CIFAR-100,
88.45 65.95 75.35 83.05 78.20SVHN, STL-10 (λ1 = 1, λ2 = 0.1)

heading: 2 RELATED WORK
text: Supervised classification techniques involve minimizing a loss function (e.g. the cross-entropy loss) to match model predictions to true labels. Unsupervised classification methods, on the other hand, learn to classify data without the use of training labels, usually with the use of clustering techniques (Bojanowski & Joulin, 2017; Dosovitskiy et al., 2014; YM. et al., 2020; Bautista et al., 2016; Caron et al., 2018; 2019; Huang et al., 2019).
More recently, new unsupervised techniques called self-supervised representation learning have been proposed. A self-supervised model learns by observing every instance of the given data and assigns its own labels, and then performs a classification task (Bojanowski & Joulin, 2017; Dosovitskiy et al., 2014; Wu et al., 2018; Dosovitskiy et al., 2016). To simplify the complexity of instancelevel classification, a memory bank (Wu et al., 2018; He et al., 2020) can be used with the help of contrastive learning (Gutmann & Hyvärinen, 2010; Hjelm et al., 2019; van den Oord et al., 2019; Grill et al., 2020). Contrastive learning (Arora et al., 2019; Tosh et al., 2021; Bachman et al., 2019) is a temperature-controlled cross-entropy loss between positive pairs of similar samples and negative pairs of dissimilar samples. Positive pairs are usually considered as multiple transformations (views) (Tian et al., 2020) of a given sample using stochastic data augmentation. SimCLR (Chen et al., 2020) shows that contrastive learning can be done without the use of a memory bank, using the samples within a batch, if we have large enough batches. SwAV (Caron et al., 2020a) uses a
mixture of contrastive learning and clustering to form a swapped prediction problem that can learn even with very small batch sizes. Finally, contrastive learning can benefit from training labels, if available with a simple modification of contrasting between samples within a class and taking samples of other classes as negatives (Khosla et al., 2020). Each of these approaches show remarkable linear classification accuracy on single-domain setups.
Extending self-supervised learning to multiple diverse domains, other than ImageNet (Russakovsky et al., 2015), is a relatively less explored topic (Wallace & Hariharan, 2020). When multiple related domains are available during training, a possible approach is to use mutual information to simultaneously encode common invariant information and domain-specific information of each image (Feng et al., 2019). In our paper, we focus on a general setup where we combine diverse unrelated domains and evaluate individual domain-specific tasks.

heading: 3 MULTI-DOMAIN SELF-SUPERVISED LEARNING WITH DOMAIN LABELS
text: In this section, we define the Multi-Domain Self-Supervised Learning (MDSSL) paradigm for D training datasets where domain labels are known. We define xdi ∈ Rr and x̃i
d ∈ Rr as two transformed views of the ith sample from the dth dataset, d ∈ {1, ..., D}. Similar to SimCLR, we use a base encoder f(.) and a two-layer MLP projection head g(.) to map a given sample into the latent space. We define the latent representations of the two views of the ith sample from the dth dataset as zdi = f(g(x d i )) ∈ Rr ′ and z̃id = f(g(x̃id)) ∈ Rr ′ where r′ is the size of each latent representation. We represent mini-batches containing 2N samples (2 views per sample) fromD datasets as a matrix X ∈ R2ND×r, whose corresponding latent representation is denoted by Z ∈ R2ND×r′ .
We then calculate a similarity matrix S ∈ R(2ND)×(2ND) that contains the exponential cosine similarity scaled by a temperature parameter τ , between all the latent representations in a given batch. The (i, j)th element of S is:
S(i,j) := exp
( 1
τ zTi zj ‖zi‖‖zj‖
) (1)
where zi ∈ Rr ′ and zj ∈ Rr ′ are the ith and jth row of Z, respectively.
Sdi represents the cosine similarity between z d i and z̃i
d. Sd,d ′
ij represents the cross-dataset cosine similarity between zdi and z d′ j where d, d ′ ∈ {1, ..., D}. MDSSL aims to solve the following optimization problem:
max θ
( 1
ND D∑ d=1 N∑ i=1 logSdi
) (2)
− λ1
( 1
2ND D∑ d=1 2N∑ i=1 log 2N∑ j=1 1j 6=iS d,d ij
) (3)
− λ2
( 1
2ND D∑ d=1 2N∑ i=1 log D∑ d′=1 2N∑ j=1 1d′ 6=dS d,d′ ij
) (4)
where θ is the set of model parameters and 0 < λ1 ≤ 1 and λ2 ≥ 0 are tunable regularization parameters. This is a three-level hierarchical loss. (2) maximizes the similarity between two transformed views (xdi and x̃i
d) in the latent space. (3) minimizes the similarity between every pair of samples (xdi and x d j ) within a dataset. (4) minimizes the similarity between pairs of samples (x d i and xd ′
j ) across different datasets. When λ1 = 1 and λ2 = 0, this optimization is simplified to SimCLR (Chen et al., 2020). Therefore, we use λ1 = 1 and λ2 = 0 as the baseline in all our experiments.
λ1 helps us control the extent to which we want to minimize the similarity within a domain. We empirically observe that relaxing λ1 from the SimCLR baseline (λ1 = 0) to a value slightly less than 1, in fact generates better structure in the latent space by clustering samples within a domain relatively closer compared to samples outside a domain (See Appendix Section A.3). λ2 should always be non-negative as we always want to minimize the agreement between samples of different
datasets. We can implement the MDSSL loss efficiently as it only involves calculating the similarity matrix S once and then selecting elements according to each term mask. As the number of datasets increases, the size of S increases, gradually increasing the running time of MDSSL.

heading: 3.1 EXPERIMENTAL SETUP
text: We use ResNet-50 (He et al., 2016) as the base encoder (f(.)) and a 2-layer MLP projection head (g(.)) for all of our experiments. For data augmentation, we use a combination of random crop, random horizontal flip, random color distortion and random Gaussian blur. In all experiments, the latent representations are in a 128-dimensional space and τ = 0.1. We optimize our loss using LARS optimizer (You et al., 2017) with a learning rate of 4 and weight decay of 10−6. We train with a batch size of 1024 and train over 48, 000 iterations. We experiment with the following datasets: CIFAR10 (Krizhevsky et al., a), CIFAR-100 (Krizhevsky et al., b), STL-10 (Coates et al.), SVHN (Netzer et al., 2011), Tiny-ImageNet (Le & Yang, 2015) and DTD (Describable Textures) (Cimpoi et al., 2014). We resize all images to 32x32 in all our experiments. We use Nvidia GeForce RTX 2080 GPUs. We measure the quality of representations using the linear evaluation protocol (Kolesnikov et al., 2019; Bachman et al., 2019; van den Oord et al., 2019) where we train a linear classifier on top of frozen MDSSL representations and compute the top-1 accuracy of each domain-specific classification task. Since we train over multiple domains, we compute the top-1 accuracy over each domain to evaluate the overall model performance.

heading: 3.2 MDSSL PERFORMANCE COMPARED TO SIMCLR BASELINE
text: In this section, we analyze the performance of MDSSL and compare it to SimCLR trained on single domains (referred to as the single-domain SimCLR) and multiple domains (referred to as the multidomain SimCLR). Table 1 summarizes our results on CIFAR-10, STL-10, SVHN and CIFAR-100. We observe that, single-domain SimCLR models generalize poorly on unseen datasets. For example, SimCLR trained on CIFAR-10 achieves 92.35% top-1 accuracy on CIFAR-10 samples but only 55.97% on SVHN samples.
We also observe that the multi-domain SimCLR model shows a degraded performance when evaluated on each individual training domain. For example, SimCLR trained on the union of samples from CIFAR-10, STL-10, SVHN and CIFAR-100 achieves 82.30% top-1 accuracy on CIFAR-10, significantly lower than the performance of the single-domain SimCLR model trained on CIFAR10. Our method, MDSSL, shows a significant improvement compared to the multi-domain SimCLR and almost matches the baseline accuracy of single-domain SimCLR models on some of the training domains. Among the average top-1 accuracy, we observe up to 25% improvement from the singledomain SimCLR and a 10% improvement from the multi-domain SimCLR (See Table 1). SimCLR would require us to train 4 different single-domain models for these datasets and therefore requires more compute, memory and time. MDSSL, being a unified model, significantly outperforms SimCLR in terms of resource utilization as shown in Table 2. This makes MDSSL an efficient solution in limited resource environments.

heading: 3.3 GENERALIZATION TO UNSEEN DATASETS
text: In this section we evaluate the generalization capacity of MDSSL to unseen domains. We consider two setups: in the first case, we use CIFAR-10, STL-10 and SVHN as training datasets (domains containing less diverse datasets as their number of classes are ≤ 10) and evaluate the model performance on unseen datasets of CIFAR-100, DTD and Tiny ImageNet (highly diverse datasets whose number of classes are > 10). In the second case, we use CIFAR-100, DTD and Tiny ImageNet as
our training datasets and assess the performances on CIFAR-10, STL-10 and SVHN. We also add results on ImageNet (250K) which contains 1000 classes, each including 250 samples resized to 32x32.
Table 3 summarizes our results. Among the single-domain SimCLR models, we observe that SimCLR trained on Tiny-ImageNet generalizes relatively better than other single-domain models since Tiny-ImageNet is comparatively larger and most diverse. However, the drop in top-1 accuracy of unseen domains from the baseline is very significant even for the single-domain SimCLR trained on Tiny-ImageNet (42% drop for SVHN). Similarly, ImageNet (250K) also poorly generalizes to unseen domains.
In our first multi-domain setup (with training datasets of CIFAR-10, STL-10 and SVHN), we observe that although these training datasets are relatively less diverse, MDSSL generalizes remarkably well on more diverse datasets like CIFAR-100, DTD and Tiny-ImageNet. MDSSL also outperforms the multi-domain SimCLR in all domains (training and unseen). We observe a similar improvement when we train MDSSL on CIFAR-100, DTD and Tiny-ImageNet and on ImageNet (250K), CIFAR-100 and SVHN. MDSSL outperforms both single and multi-domain SimCLR in terms of generalization capacity. These results highlight that MDSSL is a favorable solution that achieves good accuracy on training domains and generalizes well to unseen domains.

heading: 3.4 EFFECT OF NUMBER OF TRAINING DATASETS
text: In this section, we discuss the behavior of MDSSL as we increase the number of training domains. We train MDSSL on CIFAR-10 and CIFAR-100 (2-domain baseline). We then add STL-10, SVHN, Tiny-ImageNet and DTD datasets one by one and train MDSSL.
In Figure 2, we observe that as the number of datasets increases, the top-1 accuracy also increases and finally beats the single-domain baseline. Therefore, MDSSL benefits from training over a large number of datasets.

heading: 3.5 HYPERPARAMETER SELECTION
text: The MDSSL loss is controlled by two regularizers λ1 and λ2, as shown in Section 3. When λ1 = 1 and λ2 = 0, MDSSL boils down to our baseline, SimCLR (Chen et al., 2020). As we decrease λ1 while fixing λ2 = 0, we observe that the indistribution similarity increases (See Appendix Section A.3) and eventually, all samples show a mutual similarity of 1. Consequently, the top-1 accuracy quickly degrades from the baseline as shown in the first plot in Figure 3. This behavior can be explained by Term 3 of the MDSSL loss in Section 3 which measures the mutual similarity between all samples within a dataset. Therefore, we fix λ1 ≥ 0.9 so that it mildly increases in-distribution similarity without significantly affecting the top-1 accuracy. We utilize Term 4 of the MDSSL loss by controlling λ2, to ensure that domains are more distinguishable in the latent space. In Figure 3, the second plot shows the top-1 accuracy as we increase λ2. The top-1 accuracy rises steadily at first, and then drops at around λ2 = 0.2. This is because after a certain threshold, the in-distribution
representations become too similar which makes them harder to classify. Therefore, a good balance should be found between λ1 and λ2 such that we achieve favourable top-1 accuracy.

heading: 4 MULTI-DOMAIN SELF-SUPERVISED LEARNING WITHOUT DOMAIN LABELS
text: Most real-world multi-domain datasets are unlabelled (i.e., domain label information is not available). In this section, we develop an extension of MDSSL for such setups by identifying pseudo domain labels via a clustering approach in the latent space. As it is common in clustering, we assume the number of domains (denoted by M ) is known.
In the MDSSL loss (especially in Term 4), we need domain labels to compute pairwise similarities of samples from two different domains. To achieve this, we first treat the problem as single-domain self-supervised learning and warm up the MDSSL encoder for the first few training iterations using the optimization described in Section 3 with =λ1 = 1, λ2 = 0 and D = 1 (i.e., SimCLR training on one domain). This warm up helps us get somewhat distinguishable representations for samples between M domains and the number of iterations to warm up is determined empirically. At the end of the warm up, we cluster the latent representations of the entire multi-domain dataset into M clusters using K-Means clustering (Hartigan & Wong, 1979). Using these clusters as pseudodomain-labels, we continue training the encoder under the MDSSL loss with λ1 ≤ 1, λ2 > 0 and D = M . As the training progresses, MDSSL improves the latent structure and therefore, we recompute clusters multiple times (determined empirically) as the training progresses to ensure that improved domain labels are used.
In practice, we observe that clustering does not provide 100% accurate domain labels, especially for datasets that are distributionally similar such as CIFAR-10 and STL-10. In such cases, we propose to use a robust clustering approach coupled with MDSSL to prevent outlier clustering noise from affecting the MDSSL training. Let us consider a MDSSL encoder that is warmed up on a multidomain dataset containingM domains. We cluster the representations of this dataset intoM clusters with centroids c1, c2, . . . , cM . Before assigning pseudo-domain-labels to each representation, we first determine if they are outliers or not. If so, we ignore these samples in training MDSSL in the next round. We say a latent sample zi is not an outlier if it is significantly closer to one of the clustering centroids compared to another. Concretely, zi is not an outlier if
max
{ ‖zi − cm‖2
‖zi − cn‖2 : 1 ≤ m ≤M, 1 ≤ n ≤M
} > 1 + (5)
where ≥ 0 is defined as an outlier threshold. When is high, it means that the given sample is close to its respective centroid. When approaches 0, it indicates that the sample is almost equidistant from at least two centroids and therefore, may not be reliably clustered into one. We ignore such samples in MDSSL training. When we perform clustering for the first time, we start with = 1 and each time we repeat clustering, we decay its value exponentially such that it approaches 0 by the end of training to ensure that at the end, all samples contribute to the MDSSL training.

heading: 4.1 PERFORMANCE OF MDSSL TRAINED WITHOUT DOMAIN LABELS
text: Our experimental setup for training MDSSL without domain labels remains the same as the one we explain in Section 3.1. We perform the SimCLR warmup for 480 iterations and update the clusters every 2, 400 iterations going forward. In this section, we consider two mixtures for training datasets: (i) CIFAR-10 and SVHN (containing visually dissimilar samples), (ii) CIFAR-10, STL-10
(containing visually similar samples). In Figure 4, we plot the TSNE of CIFAR-10 and SVHN sample embeddings while using either clustering or robust clustering approaches. We observe that with clustering, we achieve a reasonable separation between domains although there are several outliers. These outliers are significantly reduced while using robust clustering as described in Equation 5. As a result, the clusters are quite well defined and easily distinguishable.
In Figure 5, we plot the top-1 accuracy of 5 training setups: single-domain SimCLR, multi-domain SimCLR, MDSSL (with domain labels), MDSSL (with clustering) and MDSSL (with robust clustering). When trained on CIFAR-10 and SVHN, we observe that MDSSL with clustering outperforms SimCLR on both datasets and on CIFAR-100 which is an unseen domain. We also observe that MDSSL with clustering seems to generalize better to CIFAR-100 compared to MDSSL. MDSSL with clustering also outperforms SimCLR when trained on CIFAR-10 and STL-10 which are more visually similar. We also observe that applying robust clustering shows an improvement on all domains including unseen domains (CIFAR-100). These results highlight that clustering is a useful approach to identify pseudo-domain-labels and when coupled with MDSSL, it helps us learn better representations for seen and unseen domains.

heading: 5 DISCUSSION
text: We propose Multi-Domain Self-Supervised Learning (MDSSL), as a unifying approach to compute self-supervised representations for a range of datasets. We support training MDSSL under two setups: with domain labels and without domain labels. We show that MDSSL achieves up to a 25% increase in top-1 accuracy with linear evaluation compared to the SimCLR baseline on a combination of CIFAR-10, STL-10, SVHN and CIFAR-100. We also show that MDSSL is significantly more efficient than SimCLR in terms of resource (time, compute and memory) utilization, generalizes better than SimCLR in multi-domain setting, and benefits from an increase in the number of training datasets. In addition, we propose two versions of clustering that can be coupled with MDSSL when training over multiple domains without the use of domain labels. MDSSL achieves good performance even under these entirely unsupervised setups. Our unified approach, MDSSL, is general-purpose, enables training on diverse multi-domain settings, and can obtain meaningful embeddings achieving state-of-the-art results both on seen (training) and unseen benchmark datasets.

heading: 6 REPRODUCIBILITY STATEMENT
text: We share our code in the supplementary materials. We also provide several implementation details to ensure reproducibility of all our experiments. In Sections 3.1 and A.5, we provide a detailed explanation of our training setup including the architecture of our encoder, optimizers, learning rate schedule and training hyperparameters. We explain the process of hyperparameter selection in Sections 3.5 and A.3.

heading: 7 ETHICS STATEMENT
text: We use only publicly available datasets which involve classification tasks on general objects, vehicles, animals, etc. To the best of our knowledge, our work does not have a negative impact on our society or any societal group. However, as with all machine learning models, MDSSL should not be used on datasets that are inherently biased or involve harmful tasks that target or affect any particular regional, cultural or societal group. Therefore, before running MDSSL, one must select datasets and downstream tasks such that they are safe and do not amplify any social biases.

heading: A APPENDIX
text: A.1 EXPERIMENTS ON NON-OBJECT-FOCUSED DATASETS
In this section we discuss the results on 3 datasets that are not object-focused i.e, EuroSAT (Helber et al., 2019), Chest X-Ray (Pneumonia) (Kermany et al., 2018) and DTD (Cimpoi et al., 2014). These datasets are understandably not generalizable to unseen domains as shown in Table 4. However, under a multi-domain setup where we combine all of these domains, MDSSL shows a modest improvement compared to multi-domain SimCLR and almost matches the single-domain baselines.
Single-Domain Training
EuroSAT 88.95 93.57 45.75 76.09 ChestXRay 85.03 95.29 46.20 75.50 DTD 86.11 93.41 50.43 76.65
Multi-Domain Training
Si m
C LR EuroSAT, ChestXRay, 86.02 94.27 46.78 75.69DTD
M D
SS L EuroSAT, ChestXRay,
87.10 94.43 50.23 77.25DTD (λ1 = 0.9, λ2 = 0.15)
A.2 COMPARING MDSSL WITH SIMCLR PRE-TRAINED ON IMAGENET
In this section, we use a pre-trained SimCLR encoder from Pytorch Lightning Bolts (Falcon et al., 2019) and train a linear classifier on several unseen datasets. We resize all images to 32x32 during linear classification to maintain consistency with the rest of our experiments. We realize this may be an unfair comparison since the encoder is pre-trained on 224x224 images. Nevertheless, we observe a significant improvement in generalization of MDSSL over SimCLR pre-trained on full-sized ImageNet on all datasets (See Table 5).
Multi-Domain Training
M D
SS L CIFAR-10, STL-10,
87.50 65.58 88.05 76.48 49.36 24.49 65.24SVHN (λ1 = 0.9, λ2 = 0.1)
M D
SS L CIFAR-100, DTD,
81.92 62.89 72.35 83.93 54.77 30.18 64.34Tiny ImageNet (λ1 = 0.9, λ2 = 0.05)
A.3 HYPERPARAMETER SELECTION
In Figure A.1, we plot the similarity matrices of class-averaged samples of MDSSL trained on CIFAR-10 and SVHN by fixing λ2 = 0 and varying λ1. As explained in Section 3, higher λ1 increases the similarity of samples within a domain. We observe that when λ1 = −1 and λ2 = 0 (SimCLR), the similarity within a domain is comparable with the similarity across domains, meaning that, domains are indistinguishable. As we increase λ1, we see that the similarity within domains increases and eventually, all samples show a mutual similarity of 1.
In Figure A.1, although similarity within domains increases, we still cannot distinguish between domains. To achieve this, we utilize Term 4 of the MDSSL loss by controlling λ2. In Figure A.2, we vary λ2 and fix λ1 = −1 (first row) and λ1 = −0.9 (second row). As λ2 increases, the similarity across domains decreases and each domain become clearly distinguishable. When λ1 = −0.9, the effect is seen even at lower values of λ2, as MDSSL learns to simultaneously increase similarity within domains while decreasing similarity across domains.
A.4 RUNNING TIME OF MDSSL
The MDSSL optimization, as discussed in Section 3, is solved by iterating over the number of training datasets (D) in each term. Therefore, as the number of datasets increases, the running time of MDSSL will increase accordingly. We use a large batch size of 1024 which also accounts for increased running time for larger datasets. In Figure A.3, we see that as the number of training datasets increases, number of training hours of MDSSL also increases.
A.5 IMPLEMENTATION DETAILS
Table 6 summarizes the entire architecture of each component of MDSSL with the filter and output dimensions for input image size 3 × 32 × 32. Our implementation of MDSSL is on PyTorch. We

heading: 1 = 0.9
text: use ResNet-50 (He et al., 2016) as the base encoder for all our experiments. Since we have multiple datasets during training, we prepare a DataLoader for each dataset and load batches of size 1024 from each dataset. We refer to these as dataset batches. When the number of training datasets is low, we concatenate all dataset batches (X) and pass it through the encoder (f(.)) and projection head (g(.)) to get Z. However, when the number of training datasets increases, X becomes too large and may require more memory to encode. In this case, we first separately encode every dataset batch and then concatenate all dataset embeddings to get Z. This trick helps us efficiently train MDSSL on 2 GPUs with 4 training datasets and a high batch size of 1024.
In Section 3.1, we discuss the experimental setup with hyperparameters for MDSSL training. We summarize these parameters in Table 7. We evaluate MDSSL using the linear evaluation protocol
(Kolesnikov et al., 2019; Bachman et al., 2019; van den Oord et al., 2019). At test time, we discard the projection head (g(.)) and keep only the ResNet encoder (f(.)). We freeze the encoder and define a trainable linear layer that maps 128-dimensional features from the encoder to class probabilities. This is our linear classifier. We train this classifier over the frozen embeddings from the ResNet encoder for 100 epochs with a batch size of 1024. We use the SGD optimizer with an initial learning rate of 0.1. We summarize all of these parameters in Table 7. We optimize the linear classifier using the cross-entropy loss and calculate the top-1 accuracy at the end of training.

Review summaries:
This paper proposes a method for multi-domain self-supervised representation learning (MDSSL).  The contrastive MDSSL training objective consists of three  terms: (1) a loss  term  that  maximizes the similarity between two transformed views of a sample in the latent space (identical to SimCLR), (2) a loss term that maximizes the similarity between every pair of samples within a dataset,  and  (3)  a loss term  that minimizes the similarity between pairs of samples  across different datasets. Loss terms (2) and (3) are weighted using hyperparameters. Authors perform experiments using common datasets including CIFAR-10, CIFAR-100,  STL-10, TinyImageNet and  SVHN. Experiments show that MDSSL outperforms (1) representations learnt by naively combining all datasets and (2) representations learnt on individual domains,  on average classification accuracy on training domains as well as held-out domains. Authors also perform  ablation studies to explore the effect of loss term hyperparameters on  downstream performance. The paper further extends the basic MDSSL method by adding an assumption that dataset labels are not available, and determine dataset labels using k-means clustering during the training process, and also explore a robust k-means clustering method. Experiments using 2 training domains show that the downstream classification performance of this unsupervised MDSSL method is comparable to original labeled MDSSL. 

 The paper presents a contrastive learning approach to self-supervised learning (SSL) that works for multiple domains.   They show that their approach generalizes better across domains than the baseline algorithm (SimCLR). Their method is basically an extension to SimCLR, which is nice because then SimCLR acts as the perfect baseline model.  They also provide an approach to using multiple domains when the domain labels are not available (using clustering). 

The paper extends self-supervised contrastive learning from one single domain to multiple domains and achieve better generation for multi-domain performance.   Intuitively, the paper finds SimCLR, an existing self-supervised contrastive learning algorithm performs pool when takes the union of a multi-domain data (e.g., union of CIFAR10, STL-10, SVHN etc) , and is also weak when training on one specific domain and evaluates on the unobserved domain. To bridge the gap, the paper brings in an additional loss into SimCLR to mitigate the gap across different domains - maximizing the discrepancy between data from different domains.   This change achieves great improvement when training and evaluating via a unified model and better generation – train on one domain data and evaluate on the other domains.