Viewmaker Networks: Learning Views for Unsupervised Representation Learning
Authors: Alex Tamkin
Mike Wu
Noah Goodman
Venue: ICLR 2021 Poster (2021)
Keywords: unsupervised learning
self-supervised
representation learning
contrastive learning
views
data augmentation

Abstract: Many recent methods for unsupervised representation learning train models to be invariant to different "views," or distorted versions of an input. However, designing these views requires considerable trial and error by human experts, hindering widespread adoption of unsupervised representation learning methods across domains and modalities. To address this, we propose viewmaker networks: generative models that learn to produce useful views from a given input. Viewmakers are stochastic bounded adversaries: they produce views by generating and then adding an $\ell_p$-bounded perturbation to the input, and are trained adversarially with respect to the main encoder network. Remarkably, when pretraining on CIFAR-10, our learned views enable comparable transfer accuracy to the well-tuned SimCLR augmentations---despite not including transformations like cropping or color jitter. Furthermore, our learned views significantly outperform baseline augmentations on speech recordings (+9 points on average) and wearable sensor data (+17 points on average). Viewmaker views can also be combined with handcrafted views: they improve robustness to common image corruptions and can increase transfer performance in cases where handcrafted views are less explored. These results suggest that viewmakers may provide a path towards more general representation learning algorithms---reducing the domain expertise and effort needed to pretrain on a much wider set of domains. Code is available at https://github.com/alextamkin/viewmaker.

Published as a conference paper at ICLR 2021
VIEWMAKER NETWORKS: LEARNING VIEWS FOR UNSUPERVISED REPRESENTATION LEARNING
Alex Tamkin, Mike Wu, Noah Goodman Department of Computer Science
Stanford University Stanford, CA 94305, USA
{atamkin, wumike, agoodman]@stanneee
ABSTRACT
Many recent methods for unsupervised representation learning train models to be invariant to different "views," or distorted versions an input. However, design- ing these views requires considerable trial and error by human experts, hindering widespread adoption of unsupervised representation learning methods across do- mains and modalities. To address this, we propose viewmaker networks: gener- ative models that learn to produce useful views from a given input. Viewmak- ers are stochastic bounded adversaries: they produce views by generating and then adding an ly-bounded perturbation to the input, and are trained ad adversari- ally with respect the main encoder network. Remarkably, when pretraining on CIFAR-10, our learned views enable comparable transfer accuracy to the well- not including transformations like crop-
tuned SimCLR augmentations--despp ping or color jitter. Furthermore, our learned views significantly outperform base- line augmentations on speech recordings (+9 points on average) and wearable sensor data (+17 points on average). Viewmaker views can also be combined with handcrafted views: they improve robustness to common image corruptions and increase transfer performance in cases where handcrafted views are less plored. These results suggest that viewmakers may provide a path towards more general representation learning algorithms- effort needed to pretrain on a much wider set of domains. Code is available at https:: betps://qittuse com a ext amk in/ viewmaker)
the domain expertise and
Figure 1: Viewmaker networks generate complex and diverse input-dependent views for unsu- pervised learning. Examples shown are for CIFAR-10. Original image in center with pink border.
INTRODUCTION
Unsupervised representation learning has made significant recent strides, including in computer vision, where view-based methods have enabled strong performance on benchmark tasks (Wu al.
2018 Oord et al. 2018 Bachman al. 2019 Zhuang et al. 2019 Misra & Maaten 2020 He al. 2020 Chen et al. 2020a). Views here refer to human-defined data transformations, which target capabilities or invariances thought to be useful for transfer tasks. In particular, in contrastive learning of visual representations, models are trained to maximize the mutual information between different views of image, including crops, blurs, noise, and changes color and contrast (Bachman al.

---

Published as a conference paper at ICLR 2021
2019
Chen et al.
2020a). Much work has investigated the space of possible image views (and their
compositions) and understanding their effects on transfer learning (Chen et al. 2020al Wu et al. Tian et al.
2020
2019 Purushwalkam & Gupta 2020p.
The fact that views must be hand designed is a significant limitation. While views for image clas- views must be developed from scratch for new modalities. Making matters worse, even within a modality, different domains may have different optimal views (Purushwalkam & Gupta 2020). Previous studies have investigated the properties of good views through the lens of mutual information (Tian et al, 2020 Wu et al. 2020), but a croadly-applicable approach for learning views remains unstudied.
sification have been refined over many years,
In this work, we present a general method for learning diverse and useful views for contrastive learning. Rather than searching through possible compositions of existing view functions (Cubuk et al., 2018, Lim et al.) 2019), which may not be available for many modalities, our approach pro- duces views with a generative model, called the viewmaker network, trained jointly with the en- coder network. This flexibility enables learning a broad set of possible view functions, including input-dependent views, without resorting to hand-crafting or expert domain knowledge. The view- maker network is trained adversarially to create views which increase the contrastive loss of the encoder network. Rather than directly outputting views for an image, the viewmaker instead outputs stochastic perturbation that is added to the input. This perturbation is projected onto an lp sphere, controlling the effective strength of the view, similar to methods in adversarial robustness. This con- strained adversarial training method enables the model reduce the mutual information between different views while preserving useful input features for the encoder to learn from.
In summary, we contribute:
1. Viewmaker networks: to our knowledge the first modality-agnostic method to learn views for unsupervised representation learning
2. On image data, where expert-designed views have been extensively optimized, our newmaker-modeek achieve comparable transfer performance to state of the art contrastive methods while being more robust to common corruptions.
3. On speech data, our method significantly outperforms existing human-defined views on a range of speech recognition transfer tasks.
4. On time-series data from wearable sensors, our model significantly outperforms baseline views on the task of human activity recognition (e.g., cycling, running, jumping rope).
2 RELATED WORK
Unsupervised representation learning Learning useful representations from unlabeled data is fundamental problem in machine learning (Pan Yang 2009 Bengio et al., 2013). A recently successful framework for unsupervised representation learning for images invol ves training model
to be invariant to various data transformations (Bachman et al. 2019 Misra & Maaten 2020). al-
though the idea has much earlier roots (Becker & Hinton 1992 Hadsell al. 2006 Dosovitskiy et al. 2014). This idea has been expanded by a number of contrastive learning approaches which push embeddings of different views, transformed inputs, closer together, while pushing other pairs apart (Tian et al. 2019 He et al. 2020) Chen et al., 2020a b as well as non-contrastive ap- proaches which do not explicitly push apart unmatched views (Grill et al., 2020 Caron al.) 2020).
Related but more limited setups have been explored for speech, where data augmentation strategies
are less explored (Oord al. 2018 Kharitonov et 2020).
Understanding and designing views Several works have studied the role of views in contrastive learning, including from a mutual-informatioon perspective (uu et al., 2020). in relation to specific transfer tasks TTian et al.) 2019), with respect to different kinds of invariances (Purushwalkam & Gupta 2020), or via careful empirical studies (Chen et al.) 2020a). Outside of contrastive learning framework, Gontijo-Lopes et al. (2020) study how data augmentation aids generalization in vision models. Much work has explored different handcrafted data augmentation methods for supervised
learning of images (Hendrycks al. 2020 Lopes et al. 2019: Perez & Wang 2017] Yun et al.) 2019
Zhang et al. 2017). speech (Park et al. [2019] Kovács et al. 2017 Toth et al. 2018: Kharitonov et al.
2020), or in feature space (De & Taylor 2017).
2

---

Published as a conference paper at ICLR 2021
maximize
Noise
Viewmaker
minimize
Input
View
Encoder
Loss
Diagram of our method. The viewmaker network is trained to produce stochastic adver- sarial views restricted an l sphere around the input.
Figure 2:
Adversarial methods Our work is related to and inspired by work on adversarial methods, includ- ing the b balls studied in adversarial robustness (Szegedy et 2013 Madry et al.) 2017: Raghu- nathan et al. 2018) and training networks with adversarial objectives (Godrellow et al.][2014||
al. 2018p. Our work also connected the vicinal risk minimization principle (Chapelle al. and can be interpreted as producing amortized virtual adversarial examples (Miyato et al. Previous adversarial view-based pretraining methods add adversarial noise on top of existing handcrafted views (Kim et al. 2020) require access to specific transfer tasks during pretraining (Tia et al. 2020 In contrast, our method is more general: is neither specialized to particular downstream task, requires neither human-ded meenned view families. Outside of multi-view learning paradigms, adversarial methods have also seen use for representation learning in GANs (Donahue) al. 2 Donahue & Simonyan, 2019) in choosing harder negative samples (Bose al., 2018), as well as for data augmentation (Antoniou et al.) 2017]] Volpi et al., 2018: Bowles et al.) 2018). 1.18).44. Ad- versarial networks that perturb inputs have also been investigated improve GAN training (Sajjadi ct al. 2018) and to remove "shortcut" features (c.g., watermarks) for self-supervised pretext tasks
2001
2018
(Minderer al. 2020).
Learning views Outside of adversarial approaches, our work is related to other studies that seek learn data augmentation strategies by composing existing ddsinnd augmentations (Ratner
et al.
2017| Cubuk et al., 2018 Zhang et al., 2019 Ho et al. 2019: Lim et al. 2019 CCbuk et al.
2020
or by modeling variations specific to the data distribution (Tran al. 2017] Wong Kolter By contrast, our method requires no human-
2020
view functions, does not require first
pretraining a generative model, and can generate perturbations beyond naturally-ocouren variation observed in the training data (e.g. brightness or contrast), potentially conferring robustness benefits, as we explore in Section
METHOD
In contrastive learning, the objective to push embeddings of positive views (derived from the same input) close together, while pushing away embeddings of negative views (derived from different in- puts). We focus mainly on the simple, yet performant, SimCLR contrastive learning algorithm (Chen al. 2020a), but also consider memory bank-based algorithm (Wu et al.) 2018) in Section M As our method agnostic to the specific pretraining loss used, it is naturally compatible with other view-based algorithms such as MoCo (He et al. 2020). BYOL (Grill al. 2020, and SwAV CCron
et al. 2020) by similarly substituting the data transformation pipeline with a viewmaker network.
Formally, given batch of N pairs of positive views (i.j) the SimCLR loss
L
1
Σ 1((2k - 1,2 E(2k, 2 1)] where E(i.j) log km1
exp(si./T)
2N
Σ
2N k= 1 exp(six/7)
and Sa,b is the cosine similarity of the embeddings views a and b.
We generate views by perturbing examples with viewmaker network V, trained jointly with the main encoder network M. There are three attributes desirable for useful perturbations, each of which motivates aspect of method:
3

---

Published as a conference paper at ICLR 2021
1. Challenging: The perturbations should be complex and strong enough that encoder must develop useful representations to perform the supervieee task. We accomplish this by generating perturbations with neural network that is trained adversarially in- crease the loss of the encoder network. Specifically, we use a neural network that ingests the input X and outputs view X V(X).
2. Faithful:
The perturbations must not make the encoder task impossible, being so strong that they destroy all features of the input. For example, perturbations should not be able to zero out the input, making learning impossible. We accomplish this by constraining the perturbations an lp sphere around the original input. Ep constraints are common in the adversarial robustness literature where perturbations are expected to be indistinguishable In our experiments, we find the best results are achieved with an l sphere, which grants the viewmaker distortion budger that it can spend on small perturbation for a large part of the input or a more extreme perturbation for a smaller portion.
3. Stochastic: The method should be able to generate a variety of perturbations for a single input, as the encoder objective requires contrasting two different views of an input against each other. To do this, we inject random noise into the viewmaker, such that the model can learn a stochastic function that produces a different perturbed input each forward pass.
Figure 2 summarizes our method. The encoder and viewmaker are optimized in alternating steps to minimize and maximize C. respectively. We use an image-to-image neural network as our view- maker network, with an architecture adapted from work style transfer (Johnson et al. 2016). See the Appendix for more details. This network ingests the input image and outputs a perturbation that is constrained to & sphere. The sphere's 's radius is determined by the volume of the input tensor times a hyperparameter e, the distortion budget, which determines the strength of the applied perturbation. This perturbation is added to the input image and optionally clamped in the case of images to ensure all pixels are in [0, 1]. Algorithm describes this process precisely.
Algorithm 1: Generating viewmaker views
Input: Viewmaker network V. C. W H image X, l distortion budget e, noise 6 Output: Perturbed C' x W x H image X P V(X, 8) // generate perturbation
P CWH. P// // project to 4 sphere
1h
X + X + P // apply porturbation
X +
// clamp (images only)
IMAGES
We begin by applying the viewmaker to contrastive learning for images. In addition to SimCLR [Chn al. 2020a). we also consider memory bank-based instance discrimination framework Wu et al. 2018 henceforth InstDisc).
We pretrain ResNet-18 (He al. 2015) models CIFAR-10 (Krizhevsky) 2009) for 200 epochs with a batch size of 256. We train a viewmaker-encoder system with a distortion budget of e = 0.05. We tried distortion budgets e E {0.1, 0.05, 0.02) and found 0.05 work best; however, pate that further tuning would yield additional gains. As we can see in Figure D the learned views are diverse, consisting of qualitatively different kinds of perturbations and affecting different parts of the input. We compare the resulting encoder representations with a model trained with the expert views used for SimCLR, comprised of many human-derfe defined transformations targeting different kinds of invariances useful for image classification: cropping and-in---n--reiiing.... blurring, horizontal flipping, color dropping, and shifts brightness, contrast, saturation, and hue (Chen et al. 2020a).
antici-
4.1 TRANSFER RESULTS ON IMAGE CLASSIFICATION TASKS
We evaluate our models on CIFAR- 10, as well as eleven transfer tasks including MetaDataset (n- antafillou et al.] 2019). MSCOCO (Lin et al. 2014), MNIST (LeCun et al.] 1998), and FashionM- We use the standard linear evaluation protocol, which trains a logistic
NIST (Xiao al. 2017).
4

---

Published as a conference paper at ICLR 2021
SimCLR
Inst
SimCLR
InstDDsc
Dataset
Expt
Ours
Expt
Ours
Dataset
Expt
Ours
Expt
Ours
CIFAR-10
86.2
84.5
82.4
80.1
MNIST
97.1
98.7
98.7
98.9
MSCOCO
49.9
50.4
48.6
50.2
FaMNIST
88.3
91.5
89.2
91.4
CelebA (FI)
51.0
51.8
57.0
53.7
CUBirds
11.2
8.7
13.7
9.4
LSUN
56.2
55.0
56.0
55.6
VGGFlower
53.3
53.6
61.5
54.8
Aircraft
32.5
31.7
37.7
33.5
TrafficSign
96.6
94.9
98.9
94.3
DTD
30.4
28.8
29.8
29.8
Fungi
2.2
2.0
2.6
2.1
Table 1: Our learned views (Ours) enable comparable transfer performance to expert views (Expt) on CIFAR-10. Suite of transfer tasks using pretrained representations from CIFAR-IO for both the SimCLR and InstDisc pretraining setups. Numbers are percent accuracy with the exception of CelebA which is F1. FaMNIST stands for FashionMNIST.
regression on top of representations from frozen model. We apply the same views as in pretrain- ing, freezing the final viewmaker when using learned views; we apply views during validation. Table | shows our results, indicating comparable overall performance with SimCLR and InstDisc, all without the use of human-crafted view functions. This performance is noteworthy our & views cannot implement cropping- and-rescaling which was shown to be the most important view function Chen al. (2020a). We speculate that the ability of the viewmaker to implement partial masking of an image may enable similar kind of spatial information ablation as cropping.
4.1.1 COMPARISON TO RANDOM & NOISE
random noise sufficient to produce domain-agnostic views? To assess how important adversarial training is to the quality of the learned representations, we perform an ablation where we generate views by adding Gaussian noise normalized the same e 0.05 budget used in the previous section. Transfer accuracy on CIFAR-10 is significantly hurt by this ablation, reaching 52.01 % for a SimCLR model trained with random noise views compared to 84.50% for our method, demon- strating the importance of adversarial training to our method.
4.1.2 THE IMPORTANCE OF INTER-PATCH MUTUAL INFORMATION AND CROPPING VIEWS
Cropping-and-resizing has been identified crucial view function when pretraining on ImageNet (Chen et al., 2020a). However, what properties of a pretraining dataset make cropping useful? We hypothesize that such a dataset must have images whose patches have high mutual information. In other words, there must be some way for the model to identify that different patches of the same image come from the same image. While this may be true for many object or scene recognition datasets, it may be false for other important pretraining datasets, including medical or satellite im- agery, where features of interest are isolated to particular parts of the image.
To investigate this hypothesis, we modify the CIFAR-10 dataset to reduce the inter-patch mutual information by replacing each 16x16 corner of the image with the corner from another image in the training dataset (see Figure E for an example). Thus, random crops this dataset, which call CIFAR-10-Cornerr. will often contain completely unrelated information. When pretrained on expert views achieve 63.3% linear evaluation accuracy on the original CIFAR- 10 dataset, while viewmaker views achieve 68.8 %. This gap suggests that viewmaker views are less reliant on inter-patch mutual information than the expert views.
4.2 COMBINING VIEWMAKER AND HANDCRAFTED VIEWS
Can viewmakers improve performance in cases where some useful handcrafted views have already been identified? Chen et al. (2020a) show that views produced through cropping are significantly improved by a suite of color- based augmentations, which they argue prevents the network from relying solely on color statisties to perform the contrastive task. Here, show that viewmaker networks also enable strong gains when added on top of cropping and horizontal flipping views when pretraining CIFAR-10 without any domain-specifie knowledge. Alone, this subset of
5

---

Published as a conference paper at ICLR 2021
(a) CIFAR-10
(b) CIFAR-10-Comeers
Figure 3: Our learned views are still able yield useful information even when the inter-patch mutual information in dataset is low, as in Figure 3L
Views
Clean
Corrupted
Diff
Ours
84.5
71.4
-13.1
SimCLR
86.2
77.1
-9.1
Combined*
86.3
79.8
-6.5
(a) Accuracy on CIFAR- 10 and CIFAR-IO-C. "Overlap with CIFAR- 10-C corruptions.
0331 3083 0.0 6.00
0.314 0130 0128
Accuracy gain on CIFAR- 10-C by from adding our learned views atop expert views.
Figure 4: Performance of different views CIFAR-10-C corruptions. Our learned views enable solid performance in the face of unseen corruptions despite not explicitly including any blurring, contrast, brightness transformations during training, unlike the expert views. Adding our learned views on top of SimCLR yields additional gains in robust accuracy, especially on different kinds of noise corruptions and glass blurring.
handcrafted augmentations achieves 73.2% linear evaluation accuracy on CIFAR-IO. Combining these views with learned viewmaker perturbations (c 0.05) achieves 83.1 % | This suggests that viewmakers can significantly improve representation learning even in cases where some domain- specific views have already been developed.
4.3 ROBUSTNESS TO COMMON CORRUPTIONS
Image classification systems should behave robustly even when the data distribution is slightly dif- ferent from that seen during training. Does using viewmaker improve robustness against common types of corruptions not experienced at train time? To answer this, we evaluate both learned views, expert views, and their composition on the CIFAR-IO-C dataset (Hendrycks & Dietterich 2019). which assesses robustness to corruptions like snow, pixelation, and blurring. In this setting, corrup- tions are applied only test time, evaluating whether the classification system is robust to some types of corruptions to which humans are robust.
When considering methods in isolation, SimCLR augmentations result less of an accuracy drop from clean to corrupted data compared to our learned views, as shown in Table 4a This gap is expected, as the expert views overlap significantly with the CIFAR-10-C corruptions: both include blurring, brightness, and contrast transformations. Interestingly. however, when we train view- maker network while also applying expert augmentations ("Combined," Table 4a). we can further improve the robust accuracy, with notable gains noise and glass blur corruptions (Figure 4b This is noteworthy, as our learned views have no explicit overlap with the 1IFAR-10-C corruptions, unlike the expert augmentations In the Combined setting,
distortion budget of 0.01,
We did additional gains from using viewmakers top of the full, well-optimized of SimCLR augmentations.
We do notice smaller decline in contrast corruption accuracy, possibly due interactions between chang- ing pixel magnitudes and the f constraint.

---

Published as a conference paper at ICLR 2021
Expert
Ours (e)
ResNet-50, 960hr
Spec.
0.05
ResNer-18, 18, 100hr
Time
Spec.
0.05
0.1
LibriSpeech Sp. ID
95.9
90.0
LibriSpeech Sp. ID
97.1
91.6
88.3
84.0
VoxCelebl Sp. ID
8.6
10.7
VoxCeleb1 Sp. ID
5.7
7.8
12.1
9.1
AudioMNISS
80.2
88.0
AudioMNIST
31.7
63.9
93.3
87.9
Google Commands
28.3
32.6
Google Commands
27.1
31.9
47.4
41.6
Fluent Actions
30.5
42.5
Fluent Actions
29.4
32.0
41.6
37.9
Fluent Objects
36.2
50.8
Fluent Objects 37.1
40.3
47.6
47.6
Fluent Locations
62.0
68.9
Fluent Locations
59.7
63.3
66.5
68.3
Table 2: Our learned views significantly outperform existing views for speech transfer tasks. Linear evaluation accuracy for SimCLR models trained on LibriSpeech. Left: ResNet-18 + Lib- rispeech 100 hour, Right: ResNet-50 + Librispeech 960hr. "Time" refers to view functions applied in the time domain (Kharitonov et al. 2020, while "Spec." refers view functions applied directly to the spectrogram
Park et al.
2019
0.05 and 0.1 denote viewmaker distortion bounds e.
which we find works better than = 0.05, likely because combining the two augmentations at their full strength would make the learning task too difficult.
These results suggest that learned views are promising avenue for improving robustness self- supervised learning models.
SPEECH
Representation learning on speech data is an emerging and important research area, given the large amount of available unlabeled data and the increasing prevalence of speech-based human-computer interaction (Latif et al., 2020). However, compared to images, there is considerably less work on self-supervised learning and data augmentations for speech data. Thus, it is compelling setting to investigate whether viewmaker augmentations are broadly applicable across modalities.
5.1 SELF-SUPERVISED LEARNING SETUP
We adapt the contrastive learning setup from SimCLR (Chen et al., 2020a). Training proceeds largely the same as for images, but the inputs are 2D log mel spectrograms. We consider both view functions applied in the time-domain before the STFT, including noise, reverb, pitch shifts, and changes in loudness (Kharitonov et al. 2020h, as well spectral views, which involve masking or noising different parts of the spectrogram (Prk et al. 2019). To generate learned views, pass the spectrogram as input to the viewmaker. We normalize the spectrogram to mean zero and variance one before passing it through the viewmaker, and do not clamp the resulting perturbed spectrogram. See the Appendix for more details. We train on the Librispeech dataset (Panayotov et al. 2013) for 200 epochs, and display some examples of learned views in the Appendix.
5.2 SPEECH CLASSIFICATION RESULTS
We evaluate three speech classification datasets: Fluent Speech Commands (Lugosch et al. 2019). Google Speech Commands (Warden 2018). and spoken digit classification (Becker et al., 2018). as well as speaker classification on VoxCeleb (Nagrani et al. 2017) and Librispeech (Panay- otov et al. a 2015). all using the linear evaluation protocol for 100 epochs. In Table we report results with both the same distortion budget 0.05 as in the image domain, as well for comparison. Both versions significantly outperform the preexisting waveform and spectral aug- mentations, with a -9 percentage point improvement on average for the ResNet-18 (e = 0.05) viewmaker model over the best expert views. The gains for real-world tasks such as command iden- tification are compelling. One notable exception is the task of LibriSpeech speaker identification. Since LibriSpeech is the same dataset the model pretrained on, and this effect is not replicated on VoxCeleb1, the other speaker classification dataset, we suspect the model may be picking up on dataset-specific artifacts (e.g. background noise, microphone type) which may make the speaker
larger = 0.1,

---

Published as a conference paper at ICLR 2021
Spectral
Ours (c)
Dataset
With Noise
Without Noise
0.02
0.05
0.2
0.5
2.0
Pamap2
71.0
74.6
83.0
87.4
88.6
91.3
9.1
Table 3: Our learned views significantly outperform existing views for activity recognition wearable sensor data. Our method learns superior representations across large range of distortion budgets e, although budgets that are too strong prevent learning. Linear evaluation accuracy for ResNet18 models trained on Pamap2 with SimCLR. "Spectral" refers to view functions applied directly to the spectrogram (Park et al. 2019).
ID task artificially easier. An interesting possibility is that the worse performance of viewmaker views may result from the model being able to identify and ablate such spurious correlations in the spectrograms.
WEARABLE SENSOR DATA
To further validate that our method for learning views is useful across different modalities, we con- sider time-series data from wearable sensors. Wearable sensor data has a broad range of applica- tions, including health care, entertainment, and education (Lara & Labrador 2012). We specifically consider whether viewmaker views improve representation learning for the task of human activity recognition (HAR), for example identifying whether user is jumping rope, running, or cycling.
6.1 SELF-SUPERVISED LEARNING SETUP
We consider the Pamap2 dataset RRiss & Stricker 2012). a dataset of 12 different activities per- formed by 9 participants. Each activity contains 52 different time series, including heart rate, celerometer, gyroscope, and magnetometer data collected from sensors on the ankle, hand, and chest (all sampled 100Hz, except heart rate, which is sampled at approximately 9Hz). We linearly interpolate missing data, then take random 10s windows from subject recordings, using the same hiii///aliiaiooneestt splits as prior work (Moya Rueda et al. 2018). To create inputs for our model, we generate multi-channel image composed of one 32x32 log spectrogram for each sensor time- series window. Unlike speech data, we do not use the mel scale when generating the spectrogram. We then normalize the training and validation datasets by subtracting the mean and then dividing by the standard deviation of the training dataset.
We train with both our learned views and the spectral views Park et al. 2019) that were most successful in the speech domain (for multi-channel spectral masking, we apply the same randomly chosen mask to all channels). We also compare against variant of these views with spectrogram noise removed, which we find improves this baseline's performance.
6.2 SENSOR-BASED ACTIVITY RECOGNITION RESULTS
We train linear classifier on the frozen encoder representations for 50 epochs, reporting accuracy on the validation set. We sample 10k examples for each training epoch and 50k examples for validation. Our views significantly outperform spectral masking by 12.8 percentage points when using the same 0.05 as image and speech, and by 16.7 points when using larger 0.5 (Table We also find that a broad range of distortion budgets produces useful representations, although aerly-aggressivv budgets prevent learning (Table Bb. These results provide further evidence that method for learning views has broad applicability across different domains.
6.3 SEMI-SUPERVISER EXPERIMENTS
An especially important setting for self suff-super learning is domains where labeled data scarce costly to acquire. Here, show that our method enable strong performance when labels for only a single participant (Participant 1) out of seven are available. We compare simple supervised learning Participant I's labels against linear evaluation of our best pretrained model, which was
8

---

Published as a conference paper at ICLR 2021
trained on unlabeled data from all participants. The model architectures and training procedures are otherwise identical to the previous section. As Figure shows, pretraining with our method on unlabeled data enables significant gains over pure supervised learning when data is scarce, and even slightly outperforms the hand-crafted views trained on all 7 participants (cf. Table
Supervised Learning
Pretrain (Ours) & Transfer
Dataset
Participant
7 Participants
Participant
Participants
Pamap2
58.3
97.1
75.1
91.3
Table 4: Our method enables superior results in semi-supervised setting where labels for data from only one participant are available. Validation accuracy for activity recognition on Pamap2. Supervised Learning refers to training a randomly initialized model on the labeled data until convergence. Pretrain Transfer refers to training linear classifier off of the best pretrained model above. or Participants refers to the number of participants comprising the training set.
7 CONCLUSION
We introduce a method for learning views for unsupervised learning, demonstrating its effective- ness through strong performance image, speech, and wearable sensor modalities. Our novel generative model--ieewmake networks-enablne sentation learning process, as opposed relying on domain-spccifie knowledge or costly trial and error. There are many interesting avenues for future work. For example, while the & constraint simple by design, there may be other kinds of constraints that enable richer spaces of views and better performance. In addition, viewmaker networks may find use in supervised learning, for the purposes of data augmentation or improving robustness. Finally, it is interesting to consider what happens as the viewmaker networks increase in size: do we see performance gains robustness- accuracy trade-offs (Raghunathan al.) 2019)? Ultimately, our work is step towards more general self-supervised algorithms capable of pretraining on arbitrary data and domains.
us to efficiently learn views as part of the repre-
ACKNOWLEDGEMENTS
We would like thank Dan Yamins, Chengxu Zhuang, Shyamal Buch, Jesse Mu, Jared Davis, Aditi Raghunathan, Pranav Rajpurkar, Margalit Glasgow, and Jesse Michel for useful discussions and comments drafts. AT is supported by Open Phil AI Fellowship. MW is supported by the Stanford Interdisciplinary Graduate Fellowship as the Karr Family Fellow.
REFERENCES
Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEFE transactions on Computers, 100(1):90-93, 1974.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks, 2017.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp. 15535-15545, 2019.
Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356):611-133, 1992.
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Interpreting and explaining deep neural networks for classification of audio signals, 2018 88.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3578):1798-1888 2013.

---

Published as a conference paper at ICLR 2021
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: www.
wandb. com/ Software available from wandb.com.
Avishek Joey Bose, Huan Ling, and Yanshuai Cao. Adversarial contrastive estimation. 2018.
arXiv
preprint
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Ham- mers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueckert. Augmenting training data using generative adversarial networks, 2018
Gan augmentation:
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09885 09882, 2020.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In Advances in neural information processing systems, pp- 416-422, 2001.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum 2020c.
contrastive learning. arXiv preprint arXiv:2003.0m
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.08 09501, 2018.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Terrance DeVries and Graham W. Taylor. Dataset augmentation in feature space, 2017.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, pp. 10542-10552, 2019.
Jeff Donahue, Philipp Krähenbühh, and Trevor Darrell. Adversarial feature learning, arXiv preprint arXiv:1605.09782. 2016.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Diserimina- tive unsupervised feature learning with convolutional neural networks. In Advances in neural information processing systems, pp. 766-774, 2014.
WA Falcon. Pytorch lightning. Gitllub. Note: http://ghhhab.oomo///fooriiiggnp lightning, 3, 2019.
Raphael Gontijo-Lppe, Sylvia Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXhv:2002.08973, 2020.
Ian Goodfellow. Nips 2016 tutorial: Generative adversarial networks.
arXiv preprint
arXiv:1701.00166) 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning, arXiv preprint arXiv:2006.0rm 07733, 2020.
10

---

Published as a conference paper at ICLR 2021
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- fion (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729 9738, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. /2261, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of distt--iiittitttio generalization. arXiv preprint Ntti:2000..62444 2020.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pp. 2731-2741. PMLR, 2019.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 694-711, Cham, 2016. Springer International Publishing. ISBN 978-3- 419-464
Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, (ierre-Emmanue' Mazaré, Matthijs Douze, and Emmanuel Dupoux. Data augmenting contrastive learning of speech rep- resentations in the time domain. arXiv preprint arXiv:2007.00991, 2020.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning, 2020.
György Kováes, László Tóth, Dirk Van Compernolle, and Sriram Ganapathy. Increasing the ro- bustness of cnn acoustic models using autoregressive moving average spectrogram features and channel dropout. Pattern Recognition Letters, 100:44-50, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Oscar D Lara and Miguel A Labrador. A survey human activity recognition using wearable sensors. IEEE communications surveys tutorials, 15(3):1192-122) 2012.
Siddique Latif, Rajib Rana, Sara Khalifa, Raja Jurdak, Junaid Qadir, and Björn W Schuller. Deep representation learning in speech processing: Challenges, recent advances, and future trends. arXiv preprint arXiv:2001.00378 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11))2278-2324 1998.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Advances in Neural Information Processing Systems, pp. 6665 066556675,,20 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improv- ing robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint wwXiv:1906.02611. 2019.
Loren Lugosch, Mirco Ravanelli, Patrick Ignoto. Vikrant Singh Tomar, and Yoshua Bengio. Speech model pre-training for end-to-end spoken language understanding, 2019.

---

Published as a conference paper at ICLR 2021
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
Matthias Minderer, Olivier Bachem, Neil Houlsby, and Michael Tschannen. Automatic shortcut removal for self-supervised representation learning. arXiv preprint arXiv:2002.08822) 2020.
Ishan Mísra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and supervied leaming. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1993, 2018.
Fernando Moya Rueda, René Grzeszick, Gernot A Fink, Sascha Feldhorst, and Michael Ten Hom- pel. Convolutional neural networks for human activity recognition using body-worn sensors. In Informatics, volume 5, pp. 26. Multidisciplinary Digital Publishing Institute, 2018.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker identifi- cation dataset, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807.03744 2018.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 22(10))1345-1333 2009.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. pp. 5206-5210 04 2015. doi: 10. 10.1109/ICASSP2005. 7178964.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: 1904008799 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
Luis Perez and Jason Wang. The effectiveness of data augmentation image classification using deep learning. arXiv preprint arXiv:1712.04622) 2017.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: In- variances, augmentations and dataset biases. arXiv preprint arXiv:2007. 139966 2020.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial exam- ples. arXiv preprint arXiv:1801.09344 2018.
Aditi Raghumathan, Sang Michael Xie, Fanny Yang. John C Duchi, and Percy Liang. Adversarial training can hurt generalization. arXiv preprint arXiv:I90e 06032, 2019.
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn- ing to compose domain-speciff transformations for data augmentation. In Advances neural information processing systems, pp. 3236-3246, 2017.
Attila Reiss and Didier Stricker. Introducing new benchmarked dataset for activity monitoring. In 2012 16th International Symposium on Wearable Computers, pp. 108-109, IEEÉ, 2012.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, Springer, 2015.
12

---

Published as a conference paper at ICLR 2021
Mehdi SM Sajjadi, Giambattista Parascandolo, Arash Mehrjou, and Bernhard Schölkopf. Tempered adversarial networks. arXiv preprint arxiv:1802.043374 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint wrXxv:/906.05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint 10224 2020.
László Tóth, György Kováes, and Dirk Van Compernolle. A perceptually inspired data augmenta- tion method for noise robust cnn acoustic models. In International Conference on Speech and Computer, pp. 697-706. Springer, 2018.
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid. bayesian data augmenta- tion approach for learning deep models, 2017.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: dataset of datasets for learning to learn from few examples, 2019.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation, 2018.
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition, 2018.
Eric Wong and J. Zico Kolter. Learning perturbation sets for robust machine learning, 2020.
Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual in- formation in contrastive learning for visual representations. arXiv preprint 20ti::0005.31499 2020.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non-
parametric instance-level discrimination. arXiv preprint 1oXiv:/805.0199) 2018.
Chaowei Xiao, Bo Li, Jun-Jan Zhu, Warrn He, Mingyan Liu, and Dawn Song. Generating adver- sarial examples with adversarial networks, 2018
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprini arXiv: 170.07777
2017. 2017.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. Proceed- ings of the IEEE International Conference Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 170.09171 2017.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment. arXiv preprint 188. 2019.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6002-6012, 2019.
13


Reviews:
[{"review_id": "miZx15ds7RR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for automatic generation of data views for contrastive self-supervised learning of representations. The method consists of learning an adversarial perturbation model that aims to maximize the distance between the original image and its perturbed views in the space of learned representations. To avoid collapse to a completely information-destroying perturbation model, authors propose to limit the perturbation strength in terms of the $l_p$ norm of the added noise. Authors apply their method on various image, speech and wearable sensor datasets where the proposed approach provides an improvement over other methods.\n\nPaper strengths:\n1. The general idea of automating data augmentations for self-supervised learning seems very reasonable to me and adversarial training framework is a very viable option to implement these ideas.\n2. The method does show an empirical improvement over SimCLR.\n\nPaper weaknesses:\n1. Recent and relevant work of Grill et al, 2015 \"Bootstrap your own latent: A new approach to self-supervised Learning\" is not even mentioned and not used as a baseline in the experimental comparison.\n2. The particular form of additive perturbation considered in the paper seems somewhat limiting to me. For example, various rotations, contrast and saturation manipulations in the image domain are very difficult to reproduce within this framework and would result into a very high $l_p$ difference from the original image. It would be interesting to see if these transformations when added to the learned ones result into even better performance.\n3.  While I do appreciate a quite broad selection of data domains in the experiments, I think it is important to also evaluate the proposed method on ImageNet since it is a major and established benchmark with enough complexity that makes good representations really necessary for a god performance. \n\nOverall, I like the paper and would gladly increase my rating if authors could address the points above. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "gI17l27mIZJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a generative model to automatically generate data that is needed for contrastive learning, with a focus on the SimCLR framework, while the method itself is general. Experiments were conducted across multiple modalities, including image, speech and wearable sensor data. The results demonstrate the effectiveness of the proposed model as compared to data augmentation methods relying on human domain knowledge. \n\nThe paper is generally well-written with a clear introduction to the state-of-the-art, a strong motivation to fill in a gap in the literature for automating data augmentation for self-supervised representation learning, and a concise description of the proposed method. It addresses a relevant and timely problem, and the method is technically sound. Evaluation is thorough including experiments across modalities and data sets, in addition to an ablation study. The effectiveness of the proposed methods has been demonstrated. \n\nThere are several points that are yet to be clarified or discussed as detailed below. \n\nIn the abstract, it is said that the proposed method significantly outperforms baseline augmentations in speech (+9% absolute). As a number of experiments for different data sets with different hyper-parameters were conducted, it is unclear where the 9% absolute improvement comes from and this is not mentioned in the main body of the paper either.  \n\nThe viewmaker network aims at generating complex perturbations to the input data. What about generating perturbed data directly in a way similar to adversarial example generation?\n\nExperiments were conducted on a number of speech data sets for different tasks, which is nice. On the other hand, the performance in terms of accuracy seems to be fairly low for several tasks as compared to the performance obtained by state-of-the-art methods in the literature. \n- For example, on the Google Commands data set, accuracies of between 27.1% and 47.4% obtained in this paper are far below the baseline ones in the original dataset paper, which are between 82.7%-89.7% (depending on the settings), and the numbers are higher in latest works in the literature. \n- In another example, for the VoxCeleb data set, the accuracies for speaker identification are between 5.7%-12.1% in this paper, while the Top-1 accuracies are between 49.0%-80.5% for the baseline methods in the original dataset paper. Assuming Top-1 accuracy is used as the metric in this paper; otherwise Top-5 accuracy is even higher, which is good to specify in the paper as well. \n- Even though there might be differences in experimental settings and a linear classifier is used here, the gap between the results here and in the literature is too significant to make a solid conclusion based on the results, as the gain can potentially disappear when a stronger backend classifier is applied. Good to discuss this.\n- In general, the experimental settings for the speech experiments are unclear even after reading the supplementary material. \n\nNo statistical significant test or 95% confidence intervals (together with accuracy) or the like are presented for any experiments. \n\nThe reproducibility of the work could be enhanced with providing more details about the experimental settings. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "La_Vq9JpBN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Learning representations using self-supervision requires domain expertise to identify diverse transformations of the data samples that label preserving. This can be expensive and hard to obtain in many data modalities. The paper proposes to automate this by learning to generate transformations tailed to each modality and sample. Specifically, an adversarial strategy is applied to learning transformations that are close to the original view in the input space but hard to classify for the self-supervision encoder.\n\nPros:\n+ The problem is well motivated and the approach is easy to follow. \n+ Results are shown for multiple modalities with good performance including speech and time-series from wearable sensors. \n\nCons + Questions:\n+ Adversarial Data Augmentations [1] approaches can be a good baseline to compare with in addition to \"expert\" views.\n+ It will be helpful to elaborate on the claim - \"model generates perturbations beyond training data.\"\n+ N is not defined in Sec. 3.\n+ Does constraining the norm of the perturbations to limit the possible transformations to the low-frequency range. Without encouraging some structure in perturbations, it may not be possible to obtain certain transformations such as localized perturbations. This could explain the lower performance against handcrafted transformations. Would the adversarial perturbation in the latent space help to produce more semantic image transformations. \n\n[1] Zhang, Xinyu, et al. \"Adversarial autoaugment.\" arXiv preprint arXiv:1912.11188 (2019).\n\n---------\nThanks for the clarifications. I have raised my score.\n\nI agree that the method is easier and more general than methods such as Adversarial Autoaugment. It will be interesting to see how the approach generalizes to larger/complex datasets without an expert specified family of transformations or without a good generative model.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cb1RCFPXUTQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**[Summary]**\nThe authors proposed the Viewmaker, which learns to generate augmentation for contrastive learning. They show that the method achieves comparable results when applied for CIFAR-10, but significantly outperformed baseline augmentations in the speech domain and wearable sensor domain. \n\n**[Reason for rating]**\nI consider this work to be simple and effective. There might be other ways of learning augmentations for unsupervised or self-supervised learning, but the authors show one possible solution and demonstrate its effectiveness across image, speech, and wearable sensor domains. I have a few concerns as stated below, but I am leaning towards acceptance at this time. \n\n**[Pros]**\n- Propose a novel generative model that learns to augment inputs\n- insightful analysis from Sec. 4.1.1 to 4.2\n- Show the proposed method works for image, speech, and wearable sensors. \n- The proposed method is well-motivated in the abstract., where expert-designed augmentations hinder the widespread adoption of self-supervised representation learning methods across domains and modalities.\n\n**[Cons/Questions]**\n- Viewmaker cannot make augmentations like cropping-and-rescaling (as already discussed by the authors). The authors already showed that the \"Combined\"(coming proposed Viewmaker and SimCLR augmentations) is robust against CIFAR-C corruptions in Table 4(a). I wonder how will the \"Combined\" performs for datasets in Table 1 (if the authors can kindly consider running this as additional experiments, it doesn't have to be all datasets). \n- The authors claim that Viewmaker enables significant robustness against CIFAR-C corruptions, but we (readers) can only see that the Viewmakes performs the worst in Table 4(a). Perhaps this claim should be validated by comparing it with a baseline where traditional data augmentation is used. \n\n**[Minor comments]**\n- It's not clear to me what's the Expt in Table 1 at first glance. Perhaps indicate that it stands for Expert view in the table caption. \n- Table 2 was not referred to in Sec. 5.2. \n- Across the tables, Ours, Learned, and Viewmaker are used to refer to the proposed method. Can we make it consistent? \n\n---\n**[After rebuttal]**\n\nThanks for the clarifications. My score remains the same after reading through all the other reviews and the replies from the author. During the rebuttal, the authors attempt to answer many of the questions raised by the reviewers. While some of the replies are less satisfying, I still find this work to be worth to publish. I would encourage the authors incorporate the changes in the revision if the paper was accepted. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: New generative model to come up with data that is needed when doing contrastive learning. Like the fact that multiple modalities were considered and evaluated. The Viewmaker methods appears to do well on CIFAR-19 and outperforms baselines on speed and wearable domains. The reviewers praise the method for being simple, well-described and well-motivated. The main drawbacks stem from the fact that viewmaker cannot make certain types of image-specific augmentations (crop & rescale, as an example), but it's fair that the authors argue that their method is more domain-agnostic; and one can indeed add more domain-specific stuff if needed.

All in all, this seems like a solid paper with an easy to implement idea that is quite general and that has been shown to work in a variety of settings. It definitely belongs at ICLR.
Decision: Accept (Poster)