Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper is well written and the motivation is simple and clear. The conclusions are supported by sufficient experiments on various datasets and tasks, e.g.,  video and audio classification and retrieval tasks on datasets such as HMDB-51, UCF-101, DCASE2014, ESC-50 and VGG-Sound. The self-supervised pretraining is conducted on VGGSound, AudioSet, and IG65, showing the benefits of GDT in multiple source datasets. The introduced Generalized Data Transformations could potentially benefit multi-modal self-supervised learning in incorporating more transformations. 
  
I have the following comments:

1. In Section 3.1, the authors discussed contrastive audio-visual self-supervision. Can the model generalize to other supervision signals? If so, what's the limitation of the selection process generalization to more self-supervised signals?

2. This paper focuses on video and audio learning. How can this framework be generalized to video and text? The authors may share some insights in the conclusion. 

3. Will the order of the hierarchical transformation affect the feature learning process? For example, the order of t1 and t2 is swapped in Fig. 1 A.
review_title: Review for Multi-modal Self-Supervision from Generalized Data Transformations 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose to integrate a few data transformations into a generalized formulation. Similarly to the motivation of previous contrastive learning, the generalized transformation are required to learn robust representation while balancing between invariance and distinctiveness. Experiments show some validations of the proposed framework on audio-visual scenarios.

+ The authors provide a good summarization of existing contrastive augmentations and data sampling into a generalized formulation.
+ Video transformations in contrastive learning has not been carefully investigated before.
+ The raised problem of balancing (or enumerating) between distinctive vs invariant transformations is underexplored and worth studying.

- While the introduced formulation is a good wrap-up of possible contrastive augmentations, it has no practical impact until the users find the best combination through a brute-force enumeration of candidate transformations.
- I believe only the formulation is general, while their method or framework would not be generalizable to other datasets / modalities / self-tasks. With different scenarios, different combinations have to be experimented one by one.
- The experiment is done on a very specific scenario: audio-visual task, from which I believe that the main contribution of this work is more of the improvement of a specific audio-visual self-sup method, rather than a generalized formulation of the transformations.
- Minor: Quite many symbols hurt the readability. 
review_title: Lacking methodological and technical impact

final_score: rating: 7: Good paper, accept
review_content: ## Summary

The paper introduces a general framework dubbed Generalized Data Transformations (GDT) for self supervised learning. The framework is used to perform video-audio self supervised learning and analyze what kind of transformations the representations should be invariant to or on the contrary variant to thanks to a contrastive loss. The author demonstrate the effectiveness of the proposed approach by showing that the resulting learned video representations achieve very good performance on the HMDB51 and UCF101 downstream task. 

## Strengths

- Overall the paper is well written
- There are some interesting findings about the paper. I am notably thinking about the results in Table 1 indicating that it is beneficial to be variant to time reversal which demonstrate that some augmentation should actually be used as negatives rather than positives in contrastive learning.
- The final results are really good

## Weaknesses

**About the GDT formulation**: The idea of trying to have a general framework that can encompass all self supervised contrastive methods is a valuable effort. However, one feeling that I have about the GDT framework is that it brings more complexity (many notations are introduced e.g. $c$ the weights $w$, the different transformations $T$,...) than it actually brings new insights and benefits.

-  I have notably the feeling that one could have written a paper that would have put more emphasis on the interesting findings of the specific multi-modal case that is explore here (video-audio) rather than trying to fit the findings into cumbersome notations. Things might have been different if more than just the setup of video-audio had been explored to better illustrate the versatility of the proposed framework.

- Also I am questioning the generality of the framework. In particular for the multimodal case, I am unsure that the actual mathematical formulation works as plugging $f=(f_v, f_a)$ in equation (1) does not work and I guess does not correspond to the actual thing that is done in the experiments. What is actually done is that $f$ changes depending on the transformation (it becomes $f_a$ if the transformation corresponds to extracting the audio and $f_v$ otherwise), but this does not seem to be completely covered by the formulation. A similar issue would arise if we wanted to have different networks for different transformation of the same modality. 

In short: what are the advantages of having this framework? Did this framework helped the authors to construct new intuitions? Since it seems to be one of the main contribution of the paper this is important that the authors address that point. 

**About originality**:  If we put the introduction of the GDT aside (given the previous raised point), the paper does not bring impressive conceptual innovations for training multimodal representations as the method is similar to Korbar et al. 2018, Arandjlevovic 2017 and more recently from AVID and XDC that also learn representation by using the self supervision contained in the cross modality of video and audio. In particular the loss is not novel, the architecture used to merge the modalities are not novel and the overall conclusion is in line with previous work (that the best thing seems to be to use the other modality as an extra view for learning good representations). 

**About TR and TS invariances**

- If I understand correctly there is a single negative coming from the same video that has been time reversed (TR) (or time shifted TS), however there would be many more negatives coming from other videos (in the denominator of equation (1)). I wonder if it would be beneficial to try to upweight these single negatives coming from those specific transformations? Is this something that the authors have considered? 

- A related question is whether or not TR and TS can be combined to obtain 3 invariances? Would that be beneficial? From an intuitive point of view it seems that the two signals could be complementary. 


**Resolution of the video 112x112** 

In the appendix it is mentioned that the resolution used are 112x112. What would happen if you were to use higher input resolution (e.g. 224x224). In particular in XDC this is the resolution used and this might lead to improvements in your case as well that could further improve the performance of the method.

**Linear evaluation on UCF/HMDB**: It would be nice to also evaluate the representations on the frozen setting on UCF and HMDB (as more recent methods like ELo and MILNCE are doing). This would make the comparison in Table 2 a bit more stronger than using the retrieval or the few-shot setup that was used by methods that were not leveraging multiple modalities for learning. 

**Would the final trained models be available?** In particular the IG65M dataset is not open sourced so its important that the authors release the weights of the trained models. 

## Conclusion and assessment

Overall the paper is well written and well executed. The results are strong for self supervised learning from audio and video. Nonetheless I have some global concerns about the work, notably the limited usefulness of the introduced general framework (GDT), and the overall lack of novel concepts or new insights provided by the work (despite the TR and TS findings that seem new to me). That is why as of now I feel the paper is borderline. I am still leaning towards acceptance since I feel the paper is an important milestone for self supervised learning from video and audio but I will wait for the answers of the authors to take a final informed decision.

## Post Rebuttal comment

The authors have clarified the contributions of their work and improved the manuscript accordingly. Given this and the other positive points about the paper I am willing to increase my score to accept.


review_title: A well executed paper for audio-video self supervised learning but a potential lack of novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents Generalized Data Transformations (GDT), a framework that unifies different self-supervised learning methods. Using this framework for unsupervised video representations, the paper gets state-of-the-art results on downstream tasks.

### Strengths:
- Unification of different self-supervised methods into a single framework, which is clean and general. This includes two main very related ideas. First, the different methods are reframed to work in a contrastive setting. Second, in order to organize the combination of the methods, the authors propose a way to do it in a systematic way.
- Following the previous framework, the paper proposes combinations of self-supervised methods for video representations, and it shows better results than competing methods. Both the baselines and ablations are sensible.
- Overall well written and easy to follow.
- Code is provided.

### Weaknesses
- It is not clear whether the main contribution is the GDT framework or its application to video. After reading the experiments section, GDT feels more like a way of structuring the experiments, and less like a unified way of understanding self-supervised learning. This feeling is reinforced by the seemingly arbitrary task the authors select (audio-visual data in videos), given the method. While video representations is a very important topic, the method does not lead to video representations as its most direct application. 
- The GDT framework consists of two ideas, and they are not properly separated in the paper (or not totally unified in a single one). On the one hand, there is the formulation of different self-supervised methods as contrastive losses. On the other hand, there is the organization of data augmentation combinations. Please note that the second is not strictly necessary for the first one to work (therefore the previous point about "structuring the experiments").
- The results lack a lot of intuition and analyses. Why some data augmentations work better than others? Why opposite augmentations (being variant _and_ invariant to time shift) are useful separately? Also, the application is specific to video, so explanations and analyses of the results on video tasks should be discussed. Why do these combinations of augmentations work for video? What information are the representations encoding?
- Other unifying frameworks have been proposed for self-supervised contrastive formulations. Specifically, [1] (which is cited in the paper but not discussed) proposes to view all of these methods as multiple views of a scene (which are the positives). What conceptual contribution does this paper add on top (or instead of) the one presented in [1]?
- While the framework is theoretically clean and general, in practice there are a lot of corner cases, and exceptions. With only two augmentations (excluding the "sampling" one), a lot of the combinations are already not possible, and the authors have to propose specific ways of combining them that are very tailored to the problem. This implies that the method is general but it has a lot of hyperparameters that need to be tuned or decided manually. Similarly, the augmentations used in the paper are not systematically selected following any rules or framework, but chosen directly by the authors.

### Additional comments and questions:
- Is the reference to SimCLR incorrect? Right now it shows Tian et al 2019, in all the cases it is cited. Tian et al 2019 is the Contrastive Multiview Coding paper, not SimCLR.
- Is the format the correct one for ICLR 2021?

### Final recommendation
Overall, I believe the strengths outweigh the weaknesses and I recommend this paper to be accepted to ICLR, but I suggest the authors address the previously mentioned points.

### References
[1] Yonglong Tian, Dilip Krishnan, and Phillip Isola. _Contrastive multiview coding_. arXiv preprint arXiv:1906.05849, 2019.

review_title: Recommend accept

Meta-review: The authors provided a comprehensive rebuttal to the reviewers' feedback that addressed most of the concerns. AnonReviewer3 raised some major concerns that were partially resolved in a revision. The paper has received a split recommendation from the reviewers but within the review and discussion periods, there was no strong support towards accepting the paper. Although the paper has received some positive feedback, some of the reviewers' concerns were not fully addressed. I'd recommend the authors to address all the comments and add clarifying notes to the paper to avoid such misunderstandings if they decide to resubmit the paper to another venue.  
Decision: Reject