Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents a generative model to automatically generate data that is needed for contrastive learning, with a focus on the SimCLR framework, while the method itself is general. Experiments were conducted across multiple modalities, including image, speech and wearable sensor data. The results demonstrate the effectiveness of the proposed model as compared to data augmentation methods relying on human domain knowledge. 

The paper is generally well-written with a clear introduction to the state-of-the-art, a strong motivation to fill in a gap in the literature for automating data augmentation for self-supervised representation learning, and a concise description of the proposed method. It addresses a relevant and timely problem, and the method is technically sound. Evaluation is thorough including experiments across modalities and data sets, in addition to an ablation study. The effectiveness of the proposed methods has been demonstrated. 

There are several points that are yet to be clarified or discussed as detailed below. 

In the abstract, it is said that the proposed method significantly outperforms baseline augmentations in speech (+9% absolute). As a number of experiments for different data sets with different hyper-parameters were conducted, it is unclear where the 9% absolute improvement comes from and this is not mentioned in the main body of the paper either.  

The viewmaker network aims at generating complex perturbations to the input data. What about generating perturbed data directly in a way similar to adversarial example generation?

Experiments were conducted on a number of speech data sets for different tasks, which is nice. On the other hand, the performance in terms of accuracy seems to be fairly low for several tasks as compared to the performance obtained by state-of-the-art methods in the literature. 
- For example, on the Google Commands data set, accuracies of between 27.1% and 47.4% obtained in this paper are far below the baseline ones in the original dataset paper, which are between 82.7%-89.7% (depending on the settings), and the numbers are higher in latest works in the literature. 
- In another example, for the VoxCeleb data set, the accuracies for speaker identification are between 5.7%-12.1% in this paper, while the Top-1 accuracies are between 49.0%-80.5% for the baseline methods in the original dataset paper. Assuming Top-1 accuracy is used as the metric in this paper; otherwise Top-5 accuracy is even higher, which is good to specify in the paper as well. 
- Even though there might be differences in experimental settings and a linear classifier is used here, the gap between the results here and in the literature is too significant to make a solid conclusion based on the results, as the gain can potentially disappear when a stronger backend classifier is applied. Good to discuss this.
- In general, the experimental settings for the speech experiments are unclear even after reading the supplementary material. 

No statistical significant test or 95% confidence intervals (together with accuracy) or the like are presented for any experiments. 

The reproducibility of the work could be enhanced with providing more details about the experimental settings. 

review_title: A well-written paper on a new generative model to generate data for contrastive learning, with some deficiency in experiments

final_score: rating: 7: Good paper, accept
review_content: **[Summary]**
The authors proposed the Viewmaker, which learns to generate augmentation for contrastive learning. They show that the method achieves comparable results when applied for CIFAR-10, but significantly outperformed baseline augmentations in the speech domain and wearable sensor domain. 

**[Reason for rating]**
I consider this work to be simple and effective. There might be other ways of learning augmentations for unsupervised or self-supervised learning, but the authors show one possible solution and demonstrate its effectiveness across image, speech, and wearable sensor domains. I have a few concerns as stated below, but I am leaning towards acceptance at this time. 

**[Pros]**
- Propose a novel generative model that learns to augment inputs
- insightful analysis from Sec. 4.1.1 to 4.2
- Show the proposed method works for image, speech, and wearable sensors. 
- The proposed method is well-motivated in the abstract., where expert-designed augmentations hinder the widespread adoption of self-supervised representation learning methods across domains and modalities.

**[Cons/Questions]**
- Viewmaker cannot make augmentations like cropping-and-rescaling (as already discussed by the authors). The authors already showed that the "Combined"(coming proposed Viewmaker and SimCLR augmentations) is robust against CIFAR-C corruptions in Table 4(a). I wonder how will the "Combined" performs for datasets in Table 1 (if the authors can kindly consider running this as additional experiments, it doesn't have to be all datasets). 
- The authors claim that Viewmaker enables significant robustness against CIFAR-C corruptions, but we (readers) can only see that the Viewmakes performs the worst in Table 4(a). Perhaps this claim should be validated by comparing it with a baseline where traditional data augmentation is used. 

**[Minor comments]**
- It's not clear to me what's the Expt in Table 1 at first glance. Perhaps indicate that it stands for Expert view in the table caption. 
- Table 2 was not referred to in Sec. 5.2. 
- Across the tables, Ours, Learned, and Viewmaker are used to refer to the proposed method. Can we make it consistent? 

---
**[After rebuttal]**

Thanks for the clarifications. My score remains the same after reading through all the other reviews and the replies from the author. During the rebuttal, the authors attempt to answer many of the questions raised by the reviewers. While some of the replies are less satisfying, I still find this work to be worth to publish. I would encourage the authors incorporate the changes in the revision if the paper was accepted. 

review_title: Interesting work for learning augmentation for contrastive learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a method for automatic generation of data views for contrastive self-supervised learning of representations. The method consists of learning an adversarial perturbation model that aims to maximize the distance between the original image and its perturbed views in the space of learned representations. To avoid collapse to a completely information-destroying perturbation model, authors propose to limit the perturbation strength in terms of the $l_p$ norm of the added noise. Authors apply their method on various image, speech and wearable sensor datasets where the proposed approach provides an improvement over other methods.

Paper strengths:
1. The general idea of automating data augmentations for self-supervised learning seems very reasonable to me and adversarial training framework is a very viable option to implement these ideas.
2. The method does show an empirical improvement over SimCLR.

Paper weaknesses:
1. Recent and relevant work of Grill et al, 2015 "Bootstrap your own latent: A new approach to self-supervised Learning" is not even mentioned and not used as a baseline in the experimental comparison.
2. The particular form of additive perturbation considered in the paper seems somewhat limiting to me. For example, various rotations, contrast and saturation manipulations in the image domain are very difficult to reproduce within this framework and would result into a very high $l_p$ difference from the original image. It would be interesting to see if these transformations when added to the learned ones result into even better performance.
3.  While I do appreciate a quite broad selection of data domains in the experiments, I think it is important to also evaluate the proposed method on ImageNet since it is a major and established benchmark with enough complexity that makes good representations really necessary for a god performance. 

Overall, I like the paper and would gladly increase my rating if authors could address the points above. 
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Learning representations using self-supervision requires domain expertise to identify diverse transformations of the data samples that label preserving. This can be expensive and hard to obtain in many data modalities. The paper proposes to automate this by learning to generate transformations tailed to each modality and sample. Specifically, an adversarial strategy is applied to learning transformations that are close to the original view in the input space but hard to classify for the self-supervision encoder.

Pros:
+ The problem is well motivated and the approach is easy to follow. 
+ Results are shown for multiple modalities with good performance including speech and time-series from wearable sensors. 

Cons + Questions:
+ Adversarial Data Augmentations [1] approaches can be a good baseline to compare with in addition to "expert" views.
+ It will be helpful to elaborate on the claim - "model generates perturbations beyond training data."
+ N is not defined in Sec. 3.
+ Does constraining the norm of the perturbations to limit the possible transformations to the low-frequency range. Without encouraging some structure in perturbations, it may not be possible to obtain certain transformations such as localized perturbations. This could explain the lower performance against handcrafted transformations. Would the adversarial perturbation in the latent space help to produce more semantic image transformations. 

[1] Zhang, Xinyu, et al. "Adversarial autoaugment." arXiv preprint arXiv:1912.11188 (2019).

---------
Thanks for the clarifications. I have raised my score.

I agree that the method is easier and more general than methods such as Adversarial Autoaugment. It will be interesting to see how the approach generalizes to larger/complex datasets without an expert specified family of transformations or without a good generative model.
review_title: Well motivated approach but experimental results are underwhelming on standard image datasets 

Meta-review: New generative model to come up with data that is needed when doing contrastive learning. Like the fact that multiple modalities were considered and evaluated. The Viewmaker methods appears to do well on CIFAR-19 and outperforms baselines on speed and wearable domains. The reviewers praise the method for being simple, well-described and well-motivated. The main drawbacks stem from the fact that viewmaker cannot make certain types of image-specific augmentations (crop & rescale, as an example), but it's fair that the authors argue that their method is more domain-agnostic; and one can indeed add more domain-specific stuff if needed.

All in all, this seems like a solid paper with an easy to implement idea that is quite general and that has been shown to work in a variety of settings. It definitely belongs at ICLR.
Decision: Accept (Poster)