$i$-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
Authors: Kibok Lee
Yian Zhu
Kihyuk Sohn
Chun-Liang Li
Jinwoo Shin
Honglak Lee
Venue: ICLR 2021 Poster (2021)
Keywords: self-supervised learning
unsupervised representation learning
contrastive representation learning
data augmentation
MixUp

Abstract: Contrastive representation learning has shown to be effective to learn representations from unlabeled data. However, much progress has been made in vision domains relying on data augmentations carefully designed using domain knowledge. In this work, we propose i-Mix, a simple yet effective domain-agnostic regularization strategy for improving contrastive representation learning. We cast contrastive learning as training a non-parametric classifier by assigning a unique virtual class to each data in a batch. Then, data instances are mixed in both the input and virtual label spaces, providing more augmented data during training. In experiments, we demonstrate that i-Mix consistently improves the quality of learned representations across domains, including image, speech, and tabular data. Furthermore, we confirm its regularization effect via extensive ablation studies across model and dataset sizes. The code is available at https://github.com/kibok90/imix.

Published as a conference paper at ICLR 2021
MIX: A DOMAIN-AGNOSTIE STRATEGY FOR CONTRASTIVE REPRESENTATION LEARNING
Kibok Lee Yian Zhu Kihyuk Sohn Chun-Liang Li Jinwoo Shin' Honglak Lee
University of Michigan *Amazon Web Services "Google Cloud AI *KAIST 'LG AI Research (kibok, yianz, honglas ak famiiimmon edu *(kihyuks, chunlang/rrooggiee...... com 2 kibokkamazzo......moo amazon. com 4 "jinwooskkaist. ac.kr @honglakelgresearchhhc ai
ABSTRACT
Contrastive representation learning has shown to be effective to learn represen- tations from unlabeled data. However, much progress has been made in vision domains relying on data augmentations carefully designed using domain knowledge. In this work, we propose i-Mix, a simple yet effective domain-agnostie regulariza- tion strategy for improving contrastive representation learning. We cast contrastive learning as training non-parametric classifier by assigning a unique virtual class to each data in a batch. Then, data instances mixed in both the input and virtual label spaces, providing more augmented data during training. In experiments, we demonstrate that i-Mix consistently improves the quality of learned representations across domains, including image, speech, and tabular data. Furthermore, we con- firm its regularization effect via extensive ablation studies across model and dataset sizes. The code is available at https: //////gtttt /gut com/ kibok90 10/kk900mmmmmmmmmmmm /imix.
INTRODUCTION
Representation learning (Bengio et al., 2013) is fundamental task in machine learning since the success of machine learning relies on the quality of representation. Self-supervised representation learning (SSL) has been successfully applied in several domains, including image recognition (He et al., 2020; Chen et al., 2020a), natural language processing (Mikolov et al., 2013; Devlin et al., 2018), robotics (Sermanet et al., 2018; Lee al., 2019), speech recognition (Ravanelli et al., 2020), and video understanding (Korbar et al., 2018; Owens & Efros, 2018). Since no label is available in the unsupervised setting, pretext tasks are proposed provide self-supervision: for example, context prediction (Doersch et al., 2015), inpainting (Pathak et al., 2016), and contrastive learn- ing (Wu et al., 2018b: Hjelm al., 2019; He et al., 2020; Chen et al., 2020a). SSL has also been used as an auxiliary task to improve the performance on the main task, such as generative model learning (Chen al., 2019), eemi-supervised learning (Zhai al., 2019), and improving robustness and uncertainty (Hendrycks et al., 2019).
Recently, contrastive representation learning has gained increasing attention by showing state-of- the-art art performance in SSL for large-scale image recognition (He et al., 2020; Chen et al., 2020a), which outperforms its supervised pre-training counterpart (He et al., 2016) on downstream tasks. However, while the concept of contrastive learning is applicable to any domains, the quality of learned representations rely the domain-specific inductive bias: anchors and positive samples are obtained from the same data instance, data augmentation introduces semantically meaningful variance for better generalization. To achieve strong, yet semantically meaningful data augmentation, domain knowledge is required, e.g., color jittering in 2D images or structural information in video understanding. Hence, contrastive representation learning in different domains requires effort to develop effective data augmentations. Furthermore, while recent works have focused on large- scale settings where millions of unlabeled data is available, it would not be practical in real-world applications. For example, in lithography, acquiring data is very expensive in terms of both time and cost due to the complexity of manufacturing process (Lin et al., 2018: Sim et al., 2019).
Meanwhile, MixUp (Zhang et al., 2018) has shown to be successful data augmentation for supervised learning in various domains and tasks, including image classification (Zhang et al., 2018), generative model learning (Lucas al., 2018), and natural language processing (Guo al., 2019; Guo, 2020).

---

Published as a conference paper at ICLR 2021
In this paper, we explore the following natural, yet important question: is the idea of MixUp useful for unsupervised, self-supervised, or contrastive representation learning across different domains?
To this end, we propose instance Mix (i-Mix), domain-agnostic regularization strategy for contrastive representation learning. The key idea of i-Mix is to introduce virtual labels in batch and mix data instances and their corresponding virtual labels in the input and label spaces, respectively. We first introduce the general formulation of i-Mix, and then we show the applicability of i-Mix to state-of- the-art contrastive representation learning methods, SimCLR (Chen ct al., 2020a) and MoCo (He et al., 2020), and self-supervised learning method without negative pairs, BYO (Grill et al., 2020).
Through the experiments, we demonstrate the efficacy of i-Mix in variety of settings. First, we show the effectiveness of i-Mix by evaluating the discriminative performance of learned representations in multiple domains. Specifically, we adapt i-Mix to the contrastive representation learning methods, advancing state-of-thh the-eheate art performance across different domains, including image (Krizhevsky & Hinton, 2009; Deng et al., 2009), speech (Warden, 2018), and tabular (Asuncion & Newman, 2007) datasets. Then, we study i-Mix in various conditions, including when 1) the model and training dataset is small or large, 2) domain knowledge is limited, and 3) transfer learning.
Contribution. In summary, our contribution is three-fold:
We propose i-Mix, method for regularizing contrastive representation learning, motivated by MixUp (Zhang et al., 2018). We show how to apply i-Mix to state-of-the-art contrastive representation learning methods (Chen et al., 2020a; He et al., 2020; Grill et al., 2020).
We show that i-Mix consistently improves contrastive representation learning in both vision and non-vision domains. In particular, the discriminative performance of representations learned with i-Mix is on par with fully supervised learning on CIFAR-10/100 (Krizhevsky & Hinton, 2009) and Speech Commands (Warden, 2018).
We verify the regularization effect of i-Mix in variety of settings. We empirically observed that i-Mix significantly improves contrastive representation learning when 1) the training dataset size is small, 2) the domain knowledge for data augmentations is not enough.
2 RELATED WORK
Self-supervised representation learning (SSL) aims at learning representations from unlabeled data by solving a pretext task that derived from sulf-suuer Early works on SSL proposed pretext tasks based on data reconstruction by autoencoding (Bengio et al., 2007), such as context pre- diction (Doersch al., 2015) and inpainting (Pathak et al., 2016). Decoder-free SSL has made a huge progress in recent years. Exemplar CNN (Dosovitskiy et al., 2014) learns by classifying individual instances with data augmentations. SSL of visual representation, including colorization (Zhang et al., 2016). solving jigsaw puzzles (Noroozi & Favaro, 2016), counting the number of objects (Noroozi et al., 2017), rotation prediction (Gidaris et al., 2018) 8), next pixel prediction (Oord et al., 2018; Hénaff et al., 2019), and combinations of them (Doersch & Zisserman, 2017; Kim et al., 2018; Noroozi et al., 2018) often leverages image-specific properties to design pretext tasks. Mcanwhile, alhough deep clustering (Caron et al., 2018; 2019; Asano et al., 2020) is often distinguished from SSL, it also leverages unsupervised clustering assignments as self-supervision for representation learning.
Contrastive representation learning has gained lots of attention for SSL (He et al., 2020; Chen et al., 2020a). As opposed to early works on exemplar CNN (Dosovitskiy et al., 2014; 2015), contrastive learning maximizes similarities of positive pairs while minimizes similarities of negative pairs instead of training an instance classifier. As the choice of negative pairs is crucial for the quality of learned representations, recent works have carefully designed them. Memory-based approaches (Wu et al.. 2018b; Hjelm et al., 2019; Bachman et al.. 2019; Misra & van der Maaten, 2020; Tian et al., 2020a) maintain a memory bank of embedding vectors of instances to keep negative samples, where the memory is updated with embedding vectors extracted from previous batches. In addition, MoCo (He et al., 2020) showed that differentiating the model for anchors and positive/negative samples is effective, where the model for positive/negative samples is updated by the exponential moving average of the model for anchors. On the other hand, recent works (Ye et al., 2019; Misra & van der Maaten, 2020; Chen et al., 2020a: Tian et al., 2020a) showed that learning invariance to different views is important in contrastive representation learning. The views can be generated through data augmentations carefully designed using domain knowledge (Chen al., 2020a), splitting
2

---

Published as a conference paper at ICLR 2021
input channels (Tian et al., 2020a), or borrowing the idea of other pretext tasks, such creating jigsaw puzzles or rotating inputs (Misra & van der Maaten, 2020). In particular, SimCLR (Chen et al., 2020a) showed that simple memory-free free approach with large batch size and strong data augmentations has a comparable performance to memory-based approaches. InfoMin (Tian et al., 2020b) further studied a way generate good views for contrastive representation learning and achieved state-of-the-ee representation learning methods, BYOL (Grill et al., 2020) does not require negative pairs, where the proposed pretext task aims at predicting latent representations of one view from another. While prior works have focused on SSL on large-scale visual recognition tasks, our work focuses on contrastive representation learning in both small- and large-scale settings in different domains.
performance by combining prior works. Different from other contrastive
Data augmentation is a technique to increase the diversity of data, especially when training data are not enough for generalization. Since the augmented data must be understood as the original data, data augmentations are carefully designed using the domain knowledge about images (DeVries & Taylor, 201 7b; Cubuk et al., 2019a;b; Zhong al., 2020), speech (Amodei et al., 2016; Park et al., 2019), or natural languages (Zhang et al., 2015; Wei & Zou, 2019).
Some works have studied data augmentation with less domain knowledge: DeVries Taylor (2017a) proposed a applying augmentations in the feature space. MixUp (Zhang et al., 2018) is an effective data augmentation strategy in supervised learning, which performs vicinal risk minimization instead of empirical risk minimization, by linearly interpolating input data and their labels on the data and label spaces, respectively. On the other hand, MixUp has also shown its effectiveness in other tasks and non-vision domains, including generative adversarial networks (Lucas et al., 2018), improved robusmess and uncertainty (Hendrycks et al., 2020), and sentence classification in natural language processing (Guo, 2020; Guo et al., 2019). Other variations have also been investigated by interpolating in the feature space (Verma al., 2019) leveraging domain knowledge (Yun al., 2019). MixUp would not be directly applicable to some domains, such as point clouds, but its adaptation can be effective (Harris al., 2020). M-Mix is kind of data augmentation for better generalization in contrastive representation learning, resulting in better performances on downstream tasks.
augmentation strategy by first encoding the dataset and then
Concurrent works have leveraged the idea of MixUp for contrastive representation learning. As discussed in Section 3.3, only input data can be mixed for improving contrastive representation learning (Shen et al., 2020; Verma et al., 2020; Zhou et al., 2020), which can be considered as injecting data-driven noises. Kalantidis al. (2020) mixed hard negative samples on the embedding space. Kim et al. (2020) reported similar observations to ours but focused on small image datasets.
3 APPROACH
In this section, review MixUp (Zhang al., 2018) in supervised learning and present i-Mix in contrastive learning (He et al., 2020; Chen et al., 2020a; Grill et al., 2020). Throughout this section, let X' be data space, TR: be D-dimensional embedding space, and model f X -> RD be mapping between them. For conciseness, f. (x;) and f. (ā,, for 24, 24, E X, and model parameters are omitted in loss functions.
3.1 MIXUP IN SUPERVISED LEARNING
Suppose an one-hot label y: E {0, 1)C assigned to a data x;, where the number of classes. Let linear classifier predicting the labels consists of weight vectors (VI....,WC") where we E IRD.! Then, the cross-entropy loss for supervised learning is defined as:
exp(w]) fi)
(Tu vi) Σ Vbe log
exp(w ff)
kk=1
While the cross-entropy loss is widely used for supervised training of deep neural networks, there are several challenges of training with the cross-entropy loss, such as preventing overfitting or networks being overconfident. Several regularization techniques have been proposed to alleviate these issues, including label smoothing (Szegedy et al., 2016), adversarial training (Miyato et al., 2018), and confidence calibration (Lee et al., 2018).
We omit hias terms for presentation clarity.
3

---

Published as a conference paper at ICLR 2021
MixUp (Zhang et al., 2018) is an effective regularization with negligible computational overhead. It conducts a linear interpolation of two data instances in both input and label spaces and trains a model by minimizing the cross-entropy loss defined on the interpolated data and labels. Specifically, for two labeled data (Ii, Mt). (z5, Mj), the Mp loss is defined as follows:
MixxUp ((3:, Mi), (ry, My); A) ESup (Aæ; + (1 2)æ AB + +1 (1-A))),
Sup
(2)
where A~ Beta at a) is a mixing coefficient sampled from the beta distribution. MixUp is a vicinal risk minimization method (Chapelle al., 2001) that augments data and their labels in data-driven manner. Not only improving the generalization on the supervised task, it also improves adversarial robustness (Pang et al., 2019) and confidence calibration (Thulasidasan et al., 2019).
3.2 M-XIX IN CONTRASTIVE LEARNING
We introduce instance mix (i-Mix), a data-driven augmentation strategy for contrastive representation learning to improve the generalization of learned representations, Intuitively, instead of mixing class labels, i-Mix interpolates their virtual labels, which indicates their identity a batch.
Let B = + 3:) be a batch of data pairs, where N is the batch size, Ti, E X are two views of the same data, which are usually generated by different augmentations. For each anchor I;, we call &; and ب positive and negative samples, respectively." Then, the model f learns to maximize similarities of positive pairs (instances from the same data) while minimize similarities of negative pairs (instances from different data) in the embedding space. The output of f is 1.2-normalized, which has shown to be effective (Wu et al., 201 8a; He et al., 2020; Chen et al., 2020a). Let it {0, 1) be the virtual label of x and 2; in a batch B, where V1 = 1 and Ug = 0. For a general sample-wise contrastive loss with virtual labels {x,, v;). the i-Mix loss is defined as follows:
gi-Mix ((are, vx), vy); B, A) 1000==-MMii {Miix Mr5 A), Ao; + (1 A)oy; B),
(3)
where À ~ Beta(a, a) mixing coefficient and Mix is a mixing operator, which can be adapted
depending on target domains: for example, MixUp(x, Zy; A) λz; + (1-A)æ; (Zhang et al., 2018)
when data values are continuous, and CutMix( Ry; A) = Ox; + (1-M,)Gar (Yun et al., 2019) when data values have spatial correlation with neighbors, where M2 is binary mask filtering out region whose relative area is (1-A), and O is an element-wise multiplication. Note that some mixing operators might not work well for some domains: for example, CutMix would not be valid when data values and their spatial neighbors have no correlation. However, the MixUp operator generally works well across domains including image, speech, and tabular, use it for i-Mix formulations and experiments, unless otherwise specified. In the following, we show how to apply i-Mix to contrastive representation learning methods.
SimCLR (Chen et al., 2020a) is a simple contrastive representation learning method without a memory bank, where each anchor has one positive sample and (2N- negative samples. Let NN+i = Iz for conciseness. Then, the (2N-1)- discrimination loss is written as follows:
exp (s(Ji, fiN+O mod 2N ex)/T)
(SimCLR(1; B) = - log 2NN Ex= exp (s(Je, St))T)
(4)
where T is a temperature scaling parameter and s 1.2-normalized vectors. In this formulation, i-Mix not directly applicable because virtual labels are defined differently for each anchor. To resolve this issue, we simplify the formulation of SimCLR by excluding anchors from negative samples. Then, with virtual labels, the N-way discrimination loss is written follows:
fI is the inner product of two
exp (s(frr Fw)/T)
i pair ( (Ti, 02; B)
Cinn log
(5)
Ek=1 exp (s(r,
where we call the N-pair contrastive loss, the formulation is similar the N-pair loss in the context of metric learning (Sohn, 2016).* For two data instances (z,, v4), (zz, and batch of data
Some literature (He et al., 2020; Chen al., 2020b) refers to them query and positivecnegative keys. 3 We present the application i-Mix to the original SimCLR RR formulation in Appendix A. 'InoNNC (Oord et al., 201 8) is similar loss inspired by the idea of noise-contrastive estimation (Gutmann Hyvärinen, 2010).
4

---

Published as a conference paper at ICLR 2021
Algorithm Loss computation for i-Mix on N-pair contrastive learning in PyTorch-like style.
a, b - aug (x), aug (x) # two different views of input x
lam Beta (alpha, alpha) .sample mixing coefficient randidx = randperm (len (x))
1am * a (1-lam) * a [[randidx]
logits = matmul (normalize (model (a)), normalize (model (b) .T) /
loss lam CrossEntropyloss (logits, arange (len (x))) +
(1-lam) CrossEntropyloss (logits, randidx)
pairs B = [1 R the i-Mix loss is defined as follows:
pair ((z,, Ui), (z uy): A) LN-ppii (Xx;; (Ar; (1 A)r, du + (1 A)uy; B).
Algorithm provides the pseudocode of i-Mix on N-pair contrastive learning for one iteration."
Pair relations in contrastive loss. To use contrastive loss for representation learning, one needs to properly define pair relation {(2,, } For contrastive representation learning, where semantic class labels are not provided, the pair relation would be defined in that 1) a positive pair, d; and 7,, are different views of the same data and 2) a negative pair, x; and Tjzi, are different data instances. For supervised representation learning, Iz and T; are two data instances from the same class, while r; and Zzzzi are from different classes. Note that two augmented versions of the same data also belong to the same class, so they can also be considered as a positive pair. i-Mix is not limited to seff-supervised contrastive representation learning, but it also be used regularization method for supervised contrastive representation learning (Khosla et al., 2020) or deep metric learning (Sohn, 2016; Movshovitz-Atias al., 2017).
MoCo (He et al., 2020). In contrastive representation learning, the number of negative samples affects the quality of learned representations (Arora et al., 2019). Because SimCLR mines negative samples in the current batch, having large batch size is crucial, which often requires a lot of computational resources (Chen al., 2020a). For efficient training, recent works have maintained memory bank .M- = {Mx [ I-- which is queue of previously extracted embedding vectors, where K the size of the memory bank (Wu al., 2018b; He et al., 2020; Tian et al., 2020a;b). In addition, MoCo introduces an exponential moving average (EMA) model to extract positive and negative embedding vectors, whose parameters are updated as frw. <- mm0 + (1 - mm)0, y. where m [0, 1) is a momentum coefficient and 0 is model parameters. The loss is written as follows:
exp F/T
(M, = log
exp f(mm) Σk_ exp (s(F)
The memory bank M is then updated with [ } in the first-in first-out order. In this (K +1)-way discrimination loss, data pairs are independent to each other, such that i-Mix not directly applicable because virtual labels are defined differently for each anchor. To overcome this issue, we include the positive samples of other anchors as negative samples, similar to the N-pair contrastive loss in Eq. (5). Let 6 {0, NJ be virtual label indicating the positive sample of each anchor, where 0z =1 and 0z.zz O. Then, the (N+K)-way discrimination loss is written as follows:
N
exp (s(fe, FEXA)/T)
Σ
Ui log Σx kkml exp (s(fi, FEMA 1/T) //T) + ER exp (s(fi.Hu)/T)
As virtual labels are bounded in the same set in this formulation, i-Mix is directly applicable: for two data instances (æ,, 0z), (zy,0y), ) batch of data pairs B = {(x,, kx) B.N N and the memory bank M. the i-Mix loss is defined as follows:
gi-Mix (Ix 8.). (zy, 02)) B,); M, A) (Moc + (1 A)zy, Ao; + +1 BB (9)
For losses linear with respect to labels (e.g.. the cross cross-entropy loss), they are equivalent to Xe(Az; + (1 A)ar, vi) (1 A)E(Ar; + (1 A)ay My i.c., optimization to the mixed label is equivalent joint optimization original lahels. The proof for losses in contrastive learning methods provided in Appendix B.
5

---

Published as a conference paper at ICLR 2021
BYOL (Grill et al., 2020). Different from other contrastive representation learning methods, BYOL is a self-suue representation learning method without contrasting negative pairs. For two views of the same data 24, X, the model f learns predict view embedded with the EMA model froth from its embedding fi. Specifically, additional prediction layer g is introduced, such that the difference between gff)) and fr learned to be minimized. The BYOL loss is written as follows:
(BYOL (zi, ..) ( - A//L = 2 2. (()((f))...))
(10)
This formulation can be represented in the form of the general contrastive loss in Eq. (3), as the second view 83 can be accessed from the batch B with its virtual label e;. To derive i-Mix in BYOL, let F = [fi fllliilll................ fx fllffllllllllll III E R D be the collection of L.2 nn-normalized embedding vectors of the second views, such that ft/ fl Fu,. Then, the BYOL loss is written as follows:
|| |(Li)/|lo(()II Fv.]| 2 2- s(g(ft),Fot))
BYOL (Iz, U;; B)
(11)
For two data instances (Iz; 0i), (1zj, Uy) and batch of data pairs B = {(are, 3:) 3;) the i-Mix loss defined as follows:
Mix ((rs, 0i), (z), U;): B, >) = (Av + (1 2)0z, Au + (1 --)0;;; B).
(12)
3.3 INPUTMIX
The contribution of data augmentations the quality of learned representations is crucial in con- trastive representation learning. For the case when the domain knowledge about efficient data augmentations is limited, we propose to apply InputMix together with é-Mix, which mixes input data but not their labels. This method can be viewed as introducing structured noises driven by auxiliary data to the principal data with the largest mixing coefficient X, and the label of the principal data is assigned to the mixed data (Shen et al., 2020; Verma et al., 2020; Zhou et al., 2020). We applied InputMix and i-Mix together on image datasets Table 3.
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of i-Mix. In all experiments, we conduct contrastive representation learning on a pretext dataset and evaluate the quality of representations via supervised classification on a downstream dataset. We report the accuracy averaged over up to five runs. In the first stage, convolutional neural network (CNN) or multilayer perceptron (MLP) followed by the two-layer MLP projection head is trained on an unlabeled dataset. Then, we replace the projection head with a linear classifier and train only the linear classifier on a labeled dataset for downstream task. Except for transfer learning, datasets for the pretext and downstream tasks are the same. For i-Mix, we sample mixing coefficient x ~ Beta(a, a) for each data, where I unless otherwise stated. Additional details for the experimental settings and more experiments can be found in Appendix C.
4.1 EXPERIMENTAL SETUP
Baselines and datasets. We consider 1) N-pair contrastive learning as memory-free contrastive learning method," 2) MoCo v2 (He et al., 2020; Chen al., 2020b)* memory-based contrastive learning method, and 3) BYOL (Grill et al., 2020), which is self-supervised learning method without negative pairs. We apply i-Mix these methods and compare their performances. To show the effectiveness of i-Mix across domains, we evaluate the methods on datasets from multiple domains, including image, speech, and tabular datasets.
SIFAR-10/100 (Krizhevsky & Hinton, 2009) consist of 50k training and 10k test images, and ImageNet (Deng et al., 2009) has 1.3M training and 50k validation images, where we use them for evaluation. For ImageNet, we also use subset of randomly chosen 100 classes out of Ik classes to experiment at a different scale. We apply set of data augmentations randomly in sequence including
'Beta(a, a) is the uniform distribution when a= 1, bell-shaped when a>> 1, and bimodal when < 1. We use the N-pair formulation Eq. (5) instead SimCI. as it is simpler and more efficient to integrate i-Mix. As shown in Appendix C.2, the N-pair formulation results in no worse performance than SimCL. IR Mooo v2 improves the performance of MoCo by cosine learning schedule and more data augmentations.

---

Published as a conference paper at ICLR 2021
Domain
Dataset
N-pair
+ +-MMix
MoCo
+ +-MMi
BYOL
+i-Mix
CIFAR-10
93.3 0.1
95.6 = 0.2
93.5 = 0.2
96.1 u.111
94.2 0.2
96.3 = 0.2
Image
CIFAR-100
70.8 + 0.4
75.8
71.6 =
78.1 0.3
72.7 0.
78.6 = 0.2
Speech
Commands
94.9 0.1.
98.3
96.3
98.4
94.8 0.2
98.3 +: 0.0
Tabular
CovType
68.5 + 0.3
72.1
70.5 0.2
73.1
72.1 +.2 74.1 =
Table Comparison of contrastive representation learning methods and i-Mix in different domains.
72
Acoustcy
Epnch
201
520
13
18
90
Rether 50
Re 30
PacMo63321 301
RoWee-152
(a) CIFAR-10
(b) CIFAR-100
Figure 1: Comparison of performance gains by applying i-Mix to MoCo v2 with different model sizes and number of epochs on CIFAR-10 and 100.
random resized cropping, horizontal flipping, color jittering, gray scaling, and Gaussian blurring for ImageNet, which has shown to be effective (Chen al., 2020a:b). We use ResNet-50 (He al., 2016) as backbone network. Models are trained with a batch size of 256 (i.e., 512 including augmented data) for up to 4000 epochs on CIFAR-10 and 100, and with batch size of 512 for 800 epochs on ImageNet. For ImageNet experiments, we use the CutMix (Yun et al., 2019) version of i-Mix.
The Speech Commands dataset (Warden, 2018) 188 contains 51k training, 7k validation, and 7k test data in 12 classes. We apply of data augmentations randomly in sequence: changing amplitude, speed, and pitch in time domain, stretching, time shifting, and adding background noise frequency domain. Augmented data are then transformed with image experiments. Models are trained with a batch size of 256 for 500 epochs.
32x32 mel spectogram. We use the same architecture
For tabular dataset experiments, we consider Forest Cover Type (Cov'Type) and Higgs Boson (Higgs) from UCI repository (Asuncion & Newman, 2007). CovType contains 155 training and 566k test data in 7 classes, and Higgs contains 10.5M training and 0.5M test data for binary classification. For Higgs, we use a subset of 100k and 1M training data experiment a different seale. Since the domain knowledge for data augmentations on tabular data is limited, only masking noise with the probability 0.2 considered as data augmentation. We use a 5-layer MLP with batch normalization (loffe & Szegedy, 2015) backbone network. Models are trained with a batch size of 512 500 epochs. We use 2 CovType and Higgs as is slightly better than a
4.2 MAIN RESULTS
Table shows the wide applicability of i-Mix to state-of-the-art contrastive representation learning methods in multiple domains. i-Mix results in consistent improvements the classification accuracy, e.g., to 6.5% when i-Mix is applied to MoCo v2 on CIFAR-100 Interestingly, we observe that linear classifiers on top of representations learned with i-Mix without fine-tuning the pre-trained part often yield classification accuracy on par with simple end-to-end supervised learning from random initialization, e.g., i-Mix vs. end-to-end supervised learning performance is 96.3% vs. 95.5% on CIFAR-10, 78.6% vs. 78.9% on CIFAR- 1000 and 98.2% vs. 98.0% on Speech Commands."
4.3 REGULARIZATTIN EFFECT OF i-MIX
A better regularization method often benefits from longer training of deeper models, which is more critical when training on a small dataset. To investigate the regularization effect of i-Mix, we first
'Supervised learning with improved methods, e.g., MixUp. outperforms i-Mix. However, linear evaluation top of self-supervised representation learning proxy to measure the quality of representations learned without lahels, such that it not supposed to be compared with the performance of supervised learning.
7

---

Published as a conference paper at ICLR 2021
Domsin
Dataset
McCo v2
i-Mix
70
Method
Image
ImageNet-100
84.1
87.0
65
50
AMMx
MoCo V2
ImageNet- Ik
70.9
71.3
8 55
59
455
4.9
Domain
Dataset
MoCo v2
i-Mix
35
59
Tabular
Higgs 100k
HiggsIM
72.1 74.9
72.9
74.5
20
25
15
0.01
0.05
0.2
Relative Dataset Size
Table 2: Comparison of MoCo v2 and i-Mix on large-scale datasets.
Figure 2: Comparison of MoCo v2 and i-Mix trained the different size of ImageNet.
Aug
CIFAR-10
CIFAR-100
Speech Commands
CovType
Higgs100k
HiggsIM
1-i
4-Mix
1-Mixx
McCo
1-Mix
47.7 L1
834 181
24.7 1.3
14. 1.5
76.9 1:17
92.8 1.05
69.6 03
73.1 1n1
64.2
71.8
65.5
72.9
93.5 =02
96.1 201
21.6
78.1
96.3
98.4 00
70.5
73.1
72.1
72,9
74,9
74,5
Table 3: Comparison of MoCo v2 and i-Mix with and without data augmentations.
'InpuMix is applied when other data augmentations used.
make comparison between MoCo v2 and i-Mix by training with different model sizes and number of training epochs on the pretext task. We train ResNet-18, 50, 101, and 152 models with varying number of training epochs from 200 to 2000. Figure 1 shows the performance of MoCo v2 (solid box) and i-Mix (dashed box). The improvement by applying i-Mix to MoCo v2 is consistent over the different architecture size and the number of training epochs. Deeper models benefit from i-Mix, achieving 96.7% on CIFAR-10 10 and 79.1% on CIFAR- 100 when the backbone network is ResNet-1522 On the other hand, models trained without i-Mix start show decrease in performance, possibly due to overfitting to the pretext task when trained longer. The trend clearly shows that i-Mix results in better representations via improved regularization.
Next, study the effect of i-Mix with varying dataset sizes for the pretext tasks. Table shows the effect of i-Mix on large-scale datasets from image and tabular domains. We observe that i-Mix is particularly effective when the amount of training data is reduced, e.g., ImageNet- consists of images from 100 classes, thus has only 10% of training data compared to ImageNet- lk. However, the performance gain is reduced when the amount of training data is large. we further study representations learned with different pretext dataset sizes from 1% to 100% of the ImageNet training data in Figure Here, different from ImageNet- 100, we reduce the amount of data for each class, but maintain the number of classes the same. We observe that the performance gain by i-Mix is more significant when the size of the pretext dataset is small. Our study suggests that i-Mix is effective for regularizing self supervised representation learning when training from limited amount of data. We believe that this is aligned with findings in Zhang et al. (2018) for MixUp in supervised learning. Finally, when large-scale unlabeled dataset is available, we expect i-Mix would still be useful in obtaining better representations when trained longer with deeper and larger models.
4.4 CONTRASTIVE LEARNING WITHOUT DOMAIN- SPECIFI DATA AUGMENTATION
Data augmentations play a key role in contrastive representation learning. and therefore it raises a question when applying them to domains with limited or knowledge of such augmentations. In this section, we study the effectiveness of i-Mix as domain-agnostic strategy for contrastive representation learning, which be adapted to different domains. Table shows the performance of MoCo v2 and i-Mix with and without data augmentations. We observe significant performance gains with i-Mix when other data augmentations are not applied. For example, compared to the accuracy of 93.5% on CIFAR-10 when other data augmentations are applied, contrastive learning achieves 47.7% when trained without any data augmentations. This suggests that data augmentation an essential part for the success of contrastive representation learning (Chen et al., 2020a). However, i-Mix is able to learn meaningful representations without other data augmentations and achieves the accuracy of 83.4% on CIFAR-10.
Here, "scale" corresponds the amount data rather than image resolution.
8

---

Published as a conference paper at ICLR 2021
VOC Object Detection
ImageNet
Pretext
CIFAR-10
CIFAR-100
MoCo v2
i-Mix
Downstream
MoCo v2
+--Mix
MoCo v2
+-Mix
AP
$7.3 u.1.1
57.5 + 0.4
CIFAR-I0
93.5
96.1 +01
85.9
90.0
AP50
82.5 #
82.7 + 0.2
CIFAR-100
64.1
70.8 004
71.6
78.1
AP75
63.8 +.3
64.2 + 0.7
(a) CIFAR-I0 and 100 as the pretext dataset
(b) ImageNet as the pretext dataset
Table 4: Comparison of MoCo v2 and i-Mix in transfer learning.
In Table 3, InputMix is applied together with i-Mix to further improve the performance on image datasets. For each principal data, we mix two auxiliary data, with mixing coefficients (0.5A; + 0.5, 0.5A2, 0.5Ag), where X1, A2, )ς Dirichlet(1, 1, 1). In the above example, while i-Mix is better than baselines, adding InputMix further improves the performance of i-Mix, i.c., from 75.1% to 83,4% on CIFAR-IO, and from 50.7% to 54.0% on CIFAR-I00. This confirms that InputMix can further improve the performance when domain-specific data augmentations are not available, as discussed in Section 3.3.
Moreover, verify its effectiveness other domains beyond the image domain. For example, the performance improves from 76.9% to 92.8% on the Speech Commands dataset when we assume no other data augmentations are available. We also observe consistent improvements in accuracy for tabular datasets, even when the training dataset size is large. Although the domain knowledge for data augmentations is important achieve ofateoofthhe-ar results, our demonstration shows the potential of i-Mix to be used for a wide range of application domains where domain knowledge is particularly limited.
4.5 TRANSFERABILITY OF i-MIX
In this section, we show the improved transferability of the representations learned with i-Mix. The results are provided in Table 4. First, we train linear classifiers with downstream datasets different from the pretext dataset used to train backbone networks and evaluate their performance, e.g., CIFAR-10 as pretext and CIFAR-100 as downstream datasets vice versa. We observe consistent performance gains when learned representations from one dataset are evaluated on classification tasks of another dataset. Next, transfer representations trained on ImageNet the PASCAL VOC object detection task (Everingham et al., 2010). We follow the settings in prior works (He et al., 2020; Chen al., 2020b): the parameters of the pre-trained ResNet-50 detector with the ResNet50-C4 backbone (Ren et al., 2015), and fine-tuned end-to-end on the VOC 07+12 trainval dataset and evaluated on the VOC 07 test dataset. We report the average precision (AP) averaged over IoU thresholds between 50% to 95% at a step of 5%, and AP50 and AP76, which are AP values when IoU threshold is 50% and 75%, respectively. Similar to Table 2, we observe small but consistent performance gains in all metrics. Those results confirm that i-Mix improves the quality of learned representations, such that performances on downstream tasks are improved.
transferred
Faster R-CNN
CONCLUSION
We propose i-Mix, a domain-agnostic regularization strategy applicable to a class of self-supervised learning. The key idea of i-Mix is to introduce virtual label to each data instance, and mix both inputs and the corresponding virtual labels. We show that i-Mix is applicable to state-of-the-as self-supervised representation learning methods including SimCLR. MoCo, and BYOL, which consistently improves the performance in a variety of settings and domains. Our experimental results indicate that i-Mix is particularly effective when the training dataset size is small or data augmentation is not available, each which are prevalent in practice.
This guarantees that the mixing coefficient for the principal data is larger than 0.5 to prevent from training with noisy labels. Note that Beckham et al. (2019) also sampled mixing coefficients from the Dirichlet distribution for mixing than two data.

---

Published as a conference paper at ICLR 2021
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg. Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In ICML, 2016.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning, ICLR, 2020.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NewrlPS, 2019.
Christopher Beckham, Sina Honari, Vikas Verma, Alex M Lamb, Farnoosh Ghadiri, R Devon Hjelm, Yoshua Bengio, and Chris Pal. On adversarial mixup resynthesis. In NeurIPS, 2019.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. NIPS, 2007.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. PAMI, 3798)1198- 2013.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In ECCV, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In ICCV, 2019.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In NIPS, 2001.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby, Self-supervised gans via auxiliary rotation loss. In CVPR, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2019a.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv:1909.13771 2019b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880. 2018.
Terrance Veiries and Graham W Taylor. Dataset augmentation feature space. arXiv preprint arXiv:1702.05538, 2017a.
Terrance Drie and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04553 2017b.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In ICCV, 2017.
10

---

Published as a conference paper at ICLR 2021
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NeurIPS, 2014.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Diseriminative unsupervised feature learning with exemplar convolutional neural networks. PAMI, 38(9):1734-177,, 2015.
Mark Everingham, Luc Van Gool, Christopher KI Williams. John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2))303-338 2010.
Maurice Fréchet. Sur la distance de deux lois de probabilité. COMPTES RENDUS HEBDO- MADAIRES DES SEANCES DE I. ACADEMIE DES SCIENCES, 244(6) 1957.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In ICML, 2013.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet in hour. arXiv preprint arXiv: (70::00067777777 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07753 07733, 2020.
Hongyu Guo. Nonlinear mixup: Out-of-manifold data augmentation for text classification. In AAAI, 2020.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Augmenting data with mixup for sentence classifica- tion: An empirical study, arXiv preprint arXiv: 190,00490 2019.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, and Adam Prügel- Bennett Jonathon Hare. Fmix: Enhancing mixed sample data augmentation. arXiv preprint urXiv:2002.12044 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: 1905:0290 2019.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. NewrIPS, 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: simple data processing method to improve robustness and uncertainty. In ICLR, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In NewrlPS, 2017.
11

---

Published as a conference paper at ICLR 2021
R Devon Hjelm, Alex Fedorov, Samuel Lavoie -aroii-Mar Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NeurIPS, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.11662, 2020.
Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon. Learning image representations by completing damaged jigsaw puzzles. WACV, 2018.
Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint arXIv:2010.06300, 2020.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from sulf-supe synchronization. In NeurlPS, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. /CLR, 2018.
Michelle A Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In ICRA, 2019.
Yibo Lin, Yuki Watanabe, Taiki Kimura, Tetsuaki Matsunawa, Shigeki Nojima, Meng Li, and David Z Pan. Data efficient lithography modeling with residual neural networks and transfer learning. In Proceedings of the 2018 International Symposium Physical Design, pp. 82-89, 2018.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR. 2017.
Thomas Lucas, Corentin Tallec, Jakob Verbeek, and Yann Ollivier. Mixed batches and symmetric discriminators for gan training. In ICML, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NeurlPS, 2013.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In CVPR, 2020.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. PAMI, 41(991197991 2018.
Yair Movshovitz-Attiar Alexander Toshev, Thomas K Leung, Sergey Ioffe, and Saurabh Singh. No fuss distance metric learning using proxies. In ICCV, 2017.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In ICCV, 2017.
Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. Boosting self-supervised learning via knowledge transfer. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2021
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018.
Tianyu Pang, Kun Xu, and Jun Zhu, Mixup inference: Better exploiting mixup to defend adversarial attacks. arXiv preprint arXiv:1909.11553 2019.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint (9XYv:1904007779 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In ICASSP, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In ICRA, 2018.
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, and Trevor Darrell. Rethinking image mixture for unsupervised visual representation learning. arXiv preprint arXiv:2003. 2020.
Woojoo Sim, Kibok Lee, Dingdong Yang. Jacseung Jeong, Ji-Suk Hong, Sooryong Lee, and Honglak Lee. Automatic correction of lithography hotspots with deep generative model. In Optical Mi- crolithography XXXII, volume 10961, pp. 1096105. International Society for Optics and Photonics, 2019.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurIPS, 2016.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016
Sunil l'hulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In NeurlPS, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint 10245, 2020b.
Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii, 5(3):64-72, 1969.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
Vikas Verma, Minh- Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. Towards domain- agnostic contrastive learning. arXiv preprint 1xX0:20111.04419 2020.
13

---

Published as a conference paper at ICLR 2021
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018.
Jason W Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv: 196, 2019.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Zhengyou Zhang, and Yun Fu. Incremental classifier learning with generative adversarial networks. arXiv preprint /802.088000088.. 2018a.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018b.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In ICCV, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In ECCV, 2016.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NewrIPS, 2015.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In AAAI, 2020.
Hong- Yu Zhou, Shuang Yu, Cheng Bian, Yifan Hu, Kai Ma, and Yefeng Zheng, Comparing learn: Surpassing imagenet pretraining on radiographs by comparing image representations. In MICCAI, 2020.
14

---

Published as a conference paper at ICLR 2021
A MORE APPLICATIONS OF i-MIX
In this section, we introduce more variations of i-Mix. For conciseness, we oy denote virtual labels for different methods. We make the definition of n for each application clear.
A.1 i-MIX FOR SIMCLR
For each anchor, SimCLR takes other anchors as negative samples such that the virtual labels must be extended. Let N = ã; for conciseness, and v; {0, 10, be the virtual label indicating the positive sample of each anchor, where Mi,++i and N++ = O. Note that Us = O because the anchor itself is not counted as a positive sample. Then, Eq. (4) can be represented in the form of the cross-entropy loss:
2N
exp (s(fi.fn/TT)
(iiCLLR (244, 2y B)
Σ
U log 2N
(A.1)
2k=1, exp (s(fi, fi)/T)
The application of i-Mix to SimCLR straightforward: for two data instances (æi, vi), (zz. and a batch of data B {a, 12N the i-Mix loss is defined as follows:!2
S ((r,, vt), (my B,);
fSimCIR (Aa; + (1 - A)ay. Au + (1 - A)ey; B).
(A.2)
Note that only the input data of Eq. (A.2) is mixed, such that Sa in Eq. (A.1) is an embedding vector of the mixed data while the other f 's are the ones of clean data. Because both clean and mixed data need be fed to the network f. i-Mix for SimCLR requires twice more memory and training time compared to SimCLR when the same batch size is used.
A.2 M-MXX FOR SUPERVISED CONTRASTIVE LEARNING
Supervised contrastive learning has recently shown to be effective for supervised representation learning and it often outperforms the standard end-to-end supervised classifier learning (Khosla et al., 2020). Suppose
one-hot label yı E {0, 1] is assigned to data Tú, where C' is the number of classes. Let x N = ã; and y = y4 for conciseness. For a batch of data pairs and their labels - let n; E (0, 1)2N 2N be the virtual label indicating the positive samples of each anchor, B= (z;, 9: ) 12
where v4.j = if y; = yi and otherwise ". = Intuitively, Σj=1 Vi.j 2.N where Ny, the number of data with the label yi. Then, the supervised learning version of the SimCLR (SupCLR) loss function is written as follows:
2N
cxp (n)f
(SupCLR(25, 04; B) =
Σ n=1
oen log
(A.3)
2N
2N Σ Ek= exp (s(fs, fe)/T)
The application of i-Mix to SupCLR is straightforward: for two data instances (æ,, 0z), (æj, Uz) and a batch of data B- = {x; 2 the i-Mix loss is defined as follows:
SupCLL (<ri, uz), (Izy, vy); B, A)
ESupCER + (1 A)zz, Au; + (1-A)u;; B).
(A.4)
A.3 i-MIX FOR N-PAIR SUPERVISED CONTRASTIVE LEARNING
Note that i-Mix in Eq. (A.4) is not as efficient as SupCLR in Eq. (A.3) due to the same reason in the case of SimCLR. overcome this, we reformulate SupCLR in the form of the N-pair loss (Sohn, 2016). Suppose an one-hot label y: E {0. 1) is assigned to data Iy, where C' is the number of classes. For a batch of data pairs and their labels B= = [(2,, TG, 91) let 82 E {0, 1) be the virtual label indicating the positive samples of each anchor, where VA.j 1 if y Vi and otherwise V.jj = O. Then, the supervised version of the N-pair (Sup-N-pat pair)) contrastive loss function written
as follows:
N
exp (s(fv.
ESup-N-pair (I4, 14; B3) =
Vnn log
(A.5)
No.
EN exp (s(fi, (s(f..fi)/T)
==1
Then, the i-Mix loss for Sup-N-pair is defined as follows:
Sup N pair ((24, 24), (I,, B,); Sup--N-pair (Az, + (1 A)æ Au, + (1 A)u;: B).
(A.6)
The j-th data he excluded from the negative samples, but it does result in significant difference.
15

---

Published as a conference paper at ICLR 2021
B PROOF OF THE LINEARITY OF LOSSES WITH RESPECT TO VIRTUAL LABELS
Cross-entropy loss. The loss used in contrastive representation learning works, which is often referred to as InfoNCE (Oord et al., 2018), can be represented in the form of the cross-entropy loss as we showed for N-pair contrastive learning, SimCLR (Chen al., 2020a), and MoCo (He et al., 2020). Here we provide an example in the case of N-pair contrastive learning. Let fi f(r, + (1 A)zz) for conciseness.
pi-Mix
((xs, vi), (rg. B, Ex-pair (Ax; + (1 X)ar Ae; + (1 - A)ey; B) Nppair
exp (s(Pig:nn//T) Fn3/T)
Σ (Aeg + +(1 (1=,) log E exp (s(f2). fi)/T)
N
exp (s(ft).
fu)/T) (s(fi.fa)/T))
N
exp (s(ff).
Σ
Vi, log
- (1 - A) Σ
Ujnn log
Ek=1 exp (s(f.). f1)/T)
Σ exp (s(Jiy- fit//r)
==1
==1
>XNN-pai( (Az;; + (1 - AJzÚ, Usi B) + (1 2)ex-pair (Nzz + (1 A)az uy: Uy: B).
(B.1)
L2 loss between L22normmllied feature vectors. The BYOL (Grill et al., 2020) loss is in this type. Let F g = --s((Ar; + (1 A)rr)) (1e)))/(ll((((a) (1- A)ær))] for conciseness.
ER D such that f./llfell Fu, and
$ Mix (1zi; vz), U;); 2) = fBvoL (Ax; + (1 2)2z, A + (1 -))3))
= - F(vu; + (1 (1-))u;) =
(AFos + (1 - A) Fo)
=1-2. 1-2.9 gT (AFess (1 2))
|APes (1- Fry 2
=2 2. T (APo; (1 2)FFo,) + const
>alg0 - Foal + (1 - (1-A)llō - Foz 12 const
Afeyoo (Aæ; + (1 - A)zy, U/; B3) + (1 X)CBYOL + (1 Uy; + const.
(B.2)
Because F is not backpropagated, it can be considered as a constant.
C MORE ON EXPERIMENTS
We describe details of the experimental settings and more experimental results. For additional experiments below, we adapted the code for supervised contrastive learning (Khosla et al., 2020).
C.1
SETUP
In this section, we describe details of the experimental settings. Note that the learning rate is scaled by the batch size (Goyal et al., 2017): ScaledLeamingRatee LearningRate x BatchSize/ /556.
Image. The experiments on CIFAR-10 and 100 (Krizhevsky & Hinton, 2009) and ImageNet (Deng al., 2009) are conducted in two stages: following Chen al. (2020a), the convolutional neural network (CNN) part of ResNet-50 (He et al., 2016) followed by the two-layer multilayer perceptron (MLP) projection head (output dimensions are 2048 and 128, respectively) is trained on the unlabeled pretext dataset with batch size of 256 (i.e., 512 augmented data) with the stochastic gradient descent (SGD) optimizer with a momentum of 0.9 over up to 4000 epochs. BYOL has an additional prediction head (output dimensions are the same with the projection head), which follows the projection head, only for the model updated by gradient. 10 epochs of warmup with a linear schedule to an initial learning rate of O. 125, followed by the cosine learning rate schedule (Loshchilov & Hutter, 2017) is used. We use the weight decay of 0.0001 for the first stage. For ImageNet, we use the same hyperparameters except that the batch size is 512 and the initial learning rate is 0.03.
Phttps: / //gi thub. com/ Hobbi tlogg/ / SupContrast
"For small resolution data from CIFAR and Speech Commands, we replaced the kernal, stride, and padding size from (7,2.3) to (3, in the first convolutional layer, and removed the first max pooling layer, following Chen al. (2020a).
16

---

Published as a conference paper at ICLR 2021
Then, the head of the CNN is replaced with linear classifier, and only the linear classifier is trained with the labeled downstream dataset. For the second stage, we use a batch size of 256 with the SGD optimizer with momentum of 0.9 and an initial learning rate chosen among {1, 3, 5, 10, 30, 50, 70) over 100 epochs, where the learning rate is decayed by 0.2 after 80, 90, 95 epochs. No weight decay is used the second stage. The quality of representation is evaluated by the top-1 accuracy on the downstream task. We sample a single mixing coefficient ~ Beta(1, 1) for each training batch. The temperature is set t 0.2. Note that the optimal distribution of À and the optimal value of varies over different architectures, methods, and datasets, but the choices above result in a reasonably good performance. The memory bank size of MoCo is 65536 for ImageNet and 4096 for other datasets, and the momentum for the exponential moving average (EMA) update is 0.999 for MoCo and BOO.. We do not symmetrize the BYO loss, as does not significantly improve the performance while increasing computational complexity.
For data augmentation, we follow Chen et al. (2020a): We apply set of data augmentations randomly in sequence including resized cropping (Szegedy al., 2015), horizontal flipping with a probability of 0.5, color jittering, 15 and gray scaling with probability of 0.2. A Gaussian blurring with [0.1, applied for ImageNet. For evaluation on downstream tasks, we apply padded cropping with the pad size of 4 and horizontal flipping for CIFAR-10 and 100, and resized cropping and horizontal flipping for ImageNet.
and kernel size of 10% of the image height/width
Speech. In the experiments on Speech Commands (Warden, 2018), the network is the same with the image domain experiments, except that the number of input channels is one instead of three. The temperature is set to = 0.5 for the standard setting and = 0.2 for the augmentation setting. 10% of silence data (all zero) are added when training. At the first stage, the model is trained with the SGD optimizer with momentum of 0.9 and initial learning rate of 0.5 over 500 epochs, where the learning rate decays by 0.1 after 300 and 400 epochs and the weight decay is 0.0001. The other settings are the same with the experiments on CIFAR.
For data augmentation," we apply set of data augmentations randomly in sequence including changing amplitude, speed, and pitch in time domain, stretching, time shifting, and adding background noise in frequency domain. Each data augmentation is applied with a probability of 0.5. Augmented data are then transformed to the mel spectogram in the size of 32 x 32.
Tabular. In the experiments on CovType and Higgs (Asuncion & Newman, 2007), we take five- layer MLP with batch normalization as a backbone network. The output dimensions of layers are (2048-2048-4) 1948-4096-4096-8199
where all layers have batch normalization followed by ReLU except for the last layer. The last layer activation is maxout (Goodfellow et al., 2013) with 4 sets, such that the output dimension is 2048. On top of this five-layer MIP, we attach two-layer MLP (2048-128) as projection head. We sample a single mixing coefficient À ~ Beta (o, a) for each training batch, where a for CovType and Biggs100k: and a = for Higgs IM. The temperature is set to T = 0.1. The other settings are the same with the experiments on CIFAR, except that the batch size is 512 and the number of training epochs is 500. At the second stage, the MIP head is replaced with linear classifier. For Higgs, the classifier is computed by linear regression from the feature matrix obtained without data augmentation to the label matrix using the pscudoinverse. Since the prior knowledge on tabular data is very limited, only the masking noise with probability of 0.2 is considered data augmentation.
C.2 VARIATIONS OF i-MIX
We compare the MixUp (Zhang et al., 2018) and CutMix ( (Yun et al., 2019) variation of i-Mix on N-pair contrastive learning and SimCLR. To distinguish them, we call them ipMixUp and i-CutMix, respectively. To be fair with the memory usage in the pretext task stage, we reduce the batch size of ipMixUp and i-CutMix by half (256 to 128) for SimCLR. Following the learning rate adjustment strategy in Goyal et al. (2017), we also decrease the learning rate by half (0. (0.125 to 0.0625) when the batch size is reduced. We note that Up and i-CutMix on SimCLR take approximately 2.5 times more training time to achieve the same number of training epochs. The results are provided in Table C. We first verify that the N-pair formulation results in no worse performance than that of SimCLR. This justifies to conduct experiments using the N-pair formulation instead of that of
15 Specificallyy brightness, contrast, and saturation are scaled hy factor uniformly sampled from (0.6, 1.4] at random, and hue is rotated in the HSV space by a factor uniformly sampled from [-0.1, 0.1) at random. "ttpps: / /i thub. com tugstugi /pyterch- speech-eeehhhh commands
17

---

Published as a conference paper at ICLR 2021
N-pair i-MixUp
SimCLR
Pretext
Downstream
Vanilla
i-CutMix
Vanilla
i-MixUp
i-CutMix
CIFAR-I0
92.4 0.1
94.8 002
94.7 =.11
92.5 0.11
94.8 +
94.8 0.
CIFAR-10
CIFAR-I00
60.2 0.3
63.3 02
61.5 =0
60.0 0.2
61.4 +
57.1 = 0.4
CIFAR-IO
84.4
86.2 1.2.
85.1 0.2
84.4
85.2 0.3.
83.7 0.6
CIFAR-100
CIFAR- 100
68.7 + 0.2
72.3 0.2
72.3 0.4
68.7 0.2
72.3 0.2
71.7 =02
Table C.I:
Comparison of N-pair contrastive learning and SimCLR with i-MixUp and i-CutMix on them with ResNet-50 on CIFAR-10 and 100. We run all experiments for 1000 epochs. i-MixUp improves the accuracy on the downstream task regardless of the data distribution shift between the pretext and downstream tasks. i-CutMix shows a comparable performance with ip when the pretext and downstream datasets are the same, but it does not when the data distribution shift occurs.
Self Self--Suervised Pretext
Supervised Pretext
Pretext
Downstream
SimCLR
N-pair
+ i-Mixx
SimCLR
N-pair
+ i-Mix
CIFAR-10
92.5 1 0.1
92.4 i 0.1
94.8 0.2
95.6 103
95.7 1.1.1
97.0 a 0.1
CIFAR-10
CIFAR-100
60.0
60,2 0.3
63.3 00
58.6 0
58.9 +.5
57.8 0.6
CIFAR-100
CIFAR-10
84.4 1 0.2
84.4 1.2
86.2
0.02
86.5 1. 1.4
86.7 0.2
88.7 0.2
CIFAR-100
68.7 = 0.2
68.7 = 0.2
72.3 +
74.3 = 0.2
74.6 + 0.3
78.4
Table C.2: Comparison of the N-pair self-supervised and supervised contrastive learning methods and i-Mix on them with ResNet-50 on CIFAR-10 and 100. We also provide the performance of formulations proposed in prior works: SimCLR (Chen et al., 2020a) and its supervised version (Khosla et al., 2020). We run all experiments for 1000 epochs. i-Mix improves the accuracy on the downstream task regardless of the data distribution shift between the pretext and downstream tasks, except the case that the pretest task has smaller number of classes than that of the downstream task. The quality of representation depends on the pretext task in terms of the performance of transfer learning: self-supervised learning is better on CIFAR- 10, while supervised learning is better on CIFAR-100.
SimCLR, which is simpler and more efficient, especially when applying i-Mix, while not losing the performance. When pretext and downstream tasks share the training dataset, i-CutMix often outperforms i-MixUp, though the margin is small. However, i-CutMix shows a worse performance in transfer learning.
Table C.2 compares the performance of SimCLR. N-pair contrastive learning, and i-Mix on N-pair contrastive learning when the pretext task is self-supervised and supervised contrastive learning. We confirm that the N-pair formulation results in no worse performance than that of SimCLR in supervised contrastive learning as well. i-Mix improves the performance of supervised contrastive learning from 95.7% to 97.0% on CIFAR-10, similarly to improvement achieved by Mi for supervised learning where it improves the performance of supervised classifier learning from 95.5% to 96.6%. On the other hand, when the pretext dataset is CIFAR- 100, the performance of supervised contrastive learning is not better than that of supervised learning: MixUp improves the performance of supervised classifier learning from 78.9% to 82.2%, and i-Mix improves the performance of supervised contrastive learning from 74.6% 78.4%.
While supervised i-Mix improves the classification accuracy CIFAR-10 when trained on CIFAR- 10, the representation does not transfer well to CIFAR- 100, possibly due to overlitting to 10 class classification. When pretext dataset is CIFAR-100, supervised contrastive learning shows better performance than self-supervised contrastive learning regardless of the distribution shift, as it learss sufficiently general representation for linear classifier to work well CIFAR-10 well.
C.3 QUALITATIVE EMBEDDING ANALYSIS
Figure C.I visualizes embedding spaces learned by N-pair contrastive learning and i-Mix on CIFAR- 10 and 100. When the downstream dataset is the same with the pretext task. both contrastive learning and M-x cluster classes well, as shown in Figure C.) and Cb. However, when the downstream task is transferred CIFAR-100, i-Mix in Figure Cd) clusters classes better than contrastive
18

---

Published as a conference paper at ICLR 2021
exiomotte
Bin
1rgg
Contrastive Learning CIFAR-10
(b) i-Mix CIFAR-10
Zont
Chat
Toute
2ld1
(c) Contrastive Learning CIFAR-100
(d) i-Mix CIFAR-100
Figure C.I: t-SNE visualization of embeddings trained by contrastive learning and i-Mix with ResNet- 50 on CIFAR-I0. (a.b): Classes are well-clustered in both cases when applied to CIFAR-10. (c.d): When models are transferred to CIFAR- -0FAR-100, classes are more clustered for i-Mix than contrastive learning, as highlighted in dashed boxes. We show 10 classes for better visualization.
FED (x10 (L)
(L)
Training Ace (%) (t)
Test Acc (%) (f)
Pretext
Downstream
N-pair
+ i-Mix
N-pair
+ i-Mix
N-pair
+ i-Mix
CIFAR-10
30.0
16.7
96.1
96.1
92.4
94.8
CIFAR-10
CIFAR-100
13.8
7.9
70.7
69.5
60.2
63.3
CIFAR-10
15.2
9.7
88.1
88.8
84.4
86.2
CIFAR-100
CIFAR-100
30.4
13.3
85.6
79.0
68.7
72.3
Table C.3: Comparison of N N-pair contrastive learning and i-Mix with ResNet-50 on CIFAR- 10 and 100 in terms of the Fréchet embedding distance (FED) between training and test data distribution on the embedding space, and training and test accuracy. 1 (L) indicates that the higher (lower) number is the better. i-Mix improves contrastive learning in all metrics, which shows that i-Mix is an effective regularization method for the pretext task such that the learned representation is more generalized.
learning Figure C.I(c). Specifically, clusters of "apple," "chair," and "dolphin," can be found in Figure Cd) while they spread out in Figure C.)(c). Also, "rose" and "squirrel" are more separated in Figure C.)(d) than C.I(c). This shows that the representation learned with i-Mix is more generalizable than vanilla contrastive learning.
C.4 QUANTITATIVE EMBEDDING ANALYSIS
To estimate the quality of representation by the similarity between training and test data distribution, we measure the Fréchet embedding distance (FED): similarly to the Fréchet inception distance (FID) introduced in Heusel et al. (2017), FED is the Fréchet distance (Fréchet, 1957; Vaserstein, 1969) between the set of traiming and test embedding vectors under the Gaussian distribution assumption.
For conciseness, let f. = (IIl be an l normalized embedding vector; we normalize embedding vectors as we do when we measure the cosine similarity. Then, with the estimated mean
N "i fo and the estimated covariance S Ei (f. m) m)T, the FED can be defined as
(C.1) As shown in Table C.3, i-Mix improves FED over contrastive learning, regardless of the distribution shift. Note that the distance is large when the training dataset of the downstream task is the same with that of the pretext task. This is because the model is overfit to the training dataset, such that the distance from the test dataset, which is unseen during training, has to be large.
de d (m=*. ,S**) = =|m= -mm==-mtell m= + TT S 2(
On the other hand, Table C.3 shows that i-Mix reduces the gap between the training and test accuracy. This implies that i-Mix is an effective regularization method for pretext tasks, such that the learned representation is more generalizable downstream tasks.
19


Reviews:
[{"review_id": "a6JnRQW3szT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors present a way of framing contrastive self-supervised learning techniques that enables them to use MixUp. Results indicate this helps especially on small datasets, and in domains where the right set of data augmentations is unknown.\n\nPros:\n+ The proposed approach for adapting the effective Mix-Up strategy to contrastive learning is simple. At the same time it is elegant in its generality: it improves all self-supervised learning techniques. There have been very few such general-purpose innovations in self-supervised learning since the original instance discrimination work that started the current trend.\n+ The fact that iMix helps a lot when domain-specific data augmentations are unknown is a very significant result, since it opens up self-supervised learning to a lot of new domains.\n+ I find the experimental evaluation quite careful and detailed. I especially like that the authors went beyond images as a domain; the self-supervised learning community has been overfitting to imagenet. \n\nCons: TBH I can't think of any. This clearly seems to be a worthwhile extension to the world of self-supervised techniques, especially relevant for new domains with limited data.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qb4wXbGqeoU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe key idea of this paper is to apply MixUp-style regularization to self-supervised contrastive learning techniques (SimCLR, MoCo-v2, BYOL). This is combined with another form of MixUp that involves only the images (not the labels), but the precise nature of this component is unclear. For large networks trained on small datasets, the proposed method improves downstream classification performance by reducing overfitting (Table 1, Figure 2). The results are mixed for large-scale datasets (Table 2, Figure 3). The proposed method is also investigated as a potential domain-agnostic augmentation. Though much better than not using any augmentations at all, the results are not typically better than using standard augmentations (Table 3).\n\nStrengths:\n\nThe paper is generally clear and the figures are well-made, though the presentation could be improved here and there (see minor comments below). The question of regularizing self-supervised representation learning is an interesting one - most contrastive learning papers use such large datasets that overfitting isn't really an issue, so that topic is novel as far as I know. The proposed method is simple and effective, clearly improving the performance of several contrastive learning techniques in the regime where overfitting is an issue.\n\nWeaknesses:\n\nThe paper proposes a regularization method for contrastive learning and demonstrates that it can help to reduce overfitting. However, it has not been demonstrated that the proposed method is unusual in this regard. Perhaps simpler, more traditional regularization methods (e.g. dropout, weight decay) would have the same effect.\n\nAs far as I can tell, the proposed method actually has two components - i-Mix and InputMix - whose effects have not been separated by an ablation study.\n\nRegarding the observation that \"i-Mix + linear classifier\" can outperform supervised baselines (Section 4.2) - it would be interesting to know whether this is still true when those supervised baselines also get to benefit from a regularization technique like MixUp. It seems like that might be a more fair comparison.\n\nOverall:\n\nThe role of overfitting in contrastive self-supervised learning is an interesting and timely topic, and the proposed method is simple, effective, and general. However, additional baselines would make the importance of this particular regularization technique more apparent.\n\nMinor comments:\n\nSection 3.2: Perhaps $v_i \\in [0,1]^N$ should be $v_i \\in \\{0,1\\}^N$.\n\nSection 3.3: \n\n\"...we propose to apply InputMix together with i-Mix, which mixes input data but not their labels\" - this a somewhat confusing phrase, since the description could be read as applying to either InputMix or i-Mix. \n\nWhat does \"with the largest mixing coefficient $\\lambda$\" mean? Do we sample $\\lambda$ first, then sample another mixing coefficient on $[0, \\lambda]$?\n\nThis section is the only one where InputMix is mentioned - are we to assume that it is used in all cases where i-Mix is used?\n\nSection 4.0: It might be helpful to point out that $\\mathrm{Beta}(1,1)$ is the uniform distribution (i.e. the standard setting for MixUp). Is there any intuition behind changing the distribution of $\\lambda$ for tabular data?\n\nSection 4.1: \n\nAre the 32x32 images resized for input to the network? Or is the network modified at all for the small images?\n\nSection 4.3: \n\nIt would be nice to see experiments of this kind on ImageNet, which would allow the enormous models and longer training times to be balanced out by a suitably sized pretraining dataset.\n\nIn Figure 3, the caption says \"MoCo\" but the legend says \"N-pair\" - not sure which it should be. Also, consider including \"10% of ImageNet\" information in the caption. \n\nSection 4.3: How is this 10% ImageNet dataset constructed?\n\nSection 4.4:\n\nIt would also be helpful to explicitly compare the \"no augmentation + i-Mix\" performance against supervised numbers so the reader can easily tell whether i-Mix by itself results in a strong pretraining strategy or not.\n\nAll sections:\n\nI recommend replacing $\\beta(\\alpha, \\alpha)$ with $\\mathrm{Beta}(\\alpha, \\alpha)$ wherever it occurs in the paper.\n\nThe experiments can be a bit \"patchy\" - different self-supervised methods used for different experiments on different datasets. If possible, consider fleshing out e.g. Table 2 to cover more of the dataset/method combinations.  \n\nI would write \"MoCov2\" instead of \"MoCo\" so people don't get the wrong impression if they're just glancing through the paper. \n\n**Update: As noted elsewhere in the discussion, the authors have addressed my primary concern. I will therefore increase my score from a 6 to a 7.**", "extra_scores": {}, "extra_text": {}}, {"review_id": "PEivHdGFTbF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In essence, the premise and contribution of the paper is rather simple -- contrastive learning and mixup are both very effective methods in their respective domains, and this paper proposes the use of mixup to improve contrastive learning, and the results seem to indicate this.\n\nPositives:\n- Strong empirical results, validated on a rather wide range of settings, including dataset type (image, audio, or tabular) as well as various contrastive techniques (SimCLR, BYOL, MoCO), and ablations (dataset size and model capacity).\n\nSuggestions / negatives:\n\n- (1) I see that you experimented with CutMix [3] and that you reported some results in the supplementary material. Can you comment further on this? (Also, see (3)). I am also concerned with the potential of vanilla mixup to 'underfit' (see [1]), though perhaps this would be mitigated via careful tuning of the mixing distribution parameters 'alpha' (again, see (3)).\n- (2) Please include measures of uncertainty in your results. This is lacking in all of your tables. You mention the results are an average of up to 5 runs, but the variance should still be stated.\n- (3) Did you do any HP tuning on the alpha for Beta(alpha, alpha)? Section 4 intro says that it was fixed to 1.0 for image datasets and 2.0 for tabular. Was this the result of HP tuning? And if so, can you comment on if the choice of alpha made a big difference?\n\nExtra suggestions / comments:\n- (4) It would be nice to test the effect of mixup without modifying the contrastive learning algorithm, that way you can examine its effects purely as a data augmentation scheme. It turns out that if you use Beta(alpha, alpha+1) as your mixing distribution, it is equivalent in expectation to the mixup loss where labels are *not* mixed. This applies to the categorical cross-entropy and binary cross-entropy loss functions. See [2] for the proof of this. This means that you could simply take the algorithm you have already, make the sampling distribution Beta(alpha, alpha+1) (rather than Beta(alpha,alpha)), and then simply use CCE(L, Y) as the loss, where you are *not* mixing Y's. I'm curious to know how this works, since you would be able to test mixup in isolation without having to modify the contrastive algorithm itself. Also, note that this particular parameterisation of the mixing distribution is Beta(alpha,alpha+1) rather than Beta(alpha, alpha), and maybe it is worth seeing if it either one performs better than the other.\n\nIn summary, I think what you have here is good, but you should add measures of uncertainty to your results and also clarify how you performed the hyperparameter tuning. I'd be very interested if you address (4) as well. My rating is a 6, though I open to raising the score if my concerns have been addressed.\n\nReferences:\n\n- [1]: Guo, H., Mao, Y., & Zhang, R. (2019, July). Mixup as locally linear out-of-manifold regularization. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, pp. 3714-3722).\n- [2]: Huszár, F. (2017). Mixup: data-dependent data augmentation. Blog post: https://www.inference.vc/mixup-data-dependent-data-augmentation/\n- [3]: Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019). Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference on Computer Vision (pp. 6023-6032).", "extra_scores": {}, "extra_text": {}}, {"review_id": "MleRCDOsOBF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Introduction\nFirst off, thank you (authors) for taking the time to put this together.\n\nThe problem the authors are trying to tackle is meaningful (+1) and relevant. Contrastive learning is in dire need of removing the dependency of using transforms. This paper goes one step and even tries to generalize it across domains which is desperately needed in the field.\n\n## Paper summary\nThe authors have taken an idea from supervised learning (MixUp) and derived the matching application for contrastive learning in hopes of removing the need for domain-specific augmentations.\n\nThey show how to apply their i-Mix to SimCLR, Moco and BYOL.\n\nFinally, authors show results for vision, tabular and speech data using standard datasets for each.\n\nResults have a few sections:\n1. Evaluate cifar-10, cifar-100, Commands and CovType using Moco and BYOL in addition to i-Mix.\n2. Evaluate larger datasets, imagenet and Higgs.\n3. Evaluate how depth of a resnet and length of training are affected with i-Mix.\n4. They show ablations with models trained without their standard data augmentations and i-Mix.\n5. They evaluate transfer learning on vision (cifar-10 <=> cifar-100) and (Imagenet -> VOC detection)\n\n## Strong points.\n\n1. The overall goal they are looking at is impactful and critical.\n2. They show results on small and large datasets.\n3. They are attempting 3 domains with the same method.\n4. Their most promising result is Table 3 which shows a nice gap between models trained without data augmentation but with i-Mix instead.\n\n## Weak points\nTheir general results are not significant enough. Most results look like this (82.5 vs 82.7).\nThis hints at these results being achieved largely via hyperparameter tuning, instead of meaningfully providing an advantage.\n\n## Suggestions for improvement\n1. Show distributions for results instead of a single number. (ie: not 82.5 vs 82.7, but instead run 50-100 times and plot the histogram)\n2. Many of the ideas mentioned here are also mentioned in non-cited relevant work (for example: [YADIM](https://arxiv.org/abs/2009.00104)).\n3. Provide code. Since some of these claims come down to implementation details, it's important to see the code as well.\n\n## Recommendation\nReject", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers recommended an acceptance (rating 7) while R1 deviated much from them (rating 3). After reading R1's concerns carefully and the authors' rebuttal, I found some of the criticisms to be invalid. The authors provided a satisfactory response, addressing concerns and clarifying potential misunderstandings. Because R1 did not update the review after the rebuttal period, I am assuming the concerns have been adequately addressed. The three other reviewers all unanimously agreed that this paper tackles a timely topic, proposes a simple and effective approach, and shows convincing empirical results. I concur with the reviewers' recommendations.
Decision: Accept (Poster)