Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes and experiments with the idea of time-based augmentations: Human infants manipulate and move objects in front of themselves, move their eyes, making multiple fixations on an object, and carry objects into different rooms, providing independence of object appearance from background. All of these manipulations can be thought of as time-based augmentations of the data. Using these ideas with self-supervised learning provides a biologically-inspired way of learning representations of objects. The paper tests these ideas on three different self-supervised algorithms on four different datasets, and finds that adding time-based augmentations to standard ones significantly increases performance. The improvements are not in the 1-2% range that some papers show; rather, many results show a 10-20% increase, suggesting a real impact on self-supervised learning. Interestingly, time-based augmentations are not by themselves particularly effective (except in simulation); it is only in combination with standard augmentations that there is a substantial effect.

I have read the other reviews (which I think are substantially unfair, and/or are from reviewers who, for some reason I fail to understand, want to kill the paper), the authors' rebuttals, and skimmed the updated paper. 

My evaluation still stands: This is an excellent and exciting paper, well worthy of publication. It could be a good talk at the conference.
strength_and_weaknesses: Strengths:

+ The idea is well-motivated by data from infants
+ The improvements are substantial, and impact three different categories of self-supervised systems.
+ The analysis of the different factors affecting the outcome is fairly good.
+ The method is tested in simulation, where the kinds of interaction can be controlled, and on real-world video of object manipulations.

Weaknesses, with concrete, actionable feedback

- The idea is not completely novel; several authors have proposed similar ideas. The difference here is the thoroughness of testing, the systematic analysis, and the very large datasets used, generated via simulation.

- The paper is somewhat dense, due to all the experiments conducted, and is a little difficult to follow at times. 


clarity,_quality,_novelty_and_reproducibility: Clarity: Due to the complexity of the experiments, it is not always easy to follow what was being done.

Quality: This is a very thorough analysis of the effects of "natural" interactions on object recognition. The results are impressive.

Novelty: While temporal augmentation has been done before, this paper investigates it in a very thorough and systematic manner.

Reproducibility: the code will be made available. 

I didn't see any figures showing what the stimuli look like for 3DShapeE? Did I miss it?

p2: have shown -> have been shown
allowing to discard _> allowing the model to discard
proposed to use -> proposed using
proposes to learn -> proposes learning

p5: we approximate different kind of -> we approximate different kinds of

p6:, near top: replace the neither nor construction with either or.

p7: 2nd line from bottom: It means that -> This means that 

I'm unclear on what the "speed of object rotations" means in terms of the sequence of stimuli. Does this mean that object pairs may be in completely different orientations? A picture would help a lot here. It could be in the appendix, if necessary. 

p8: allows to encode -> allows the model to encode

encoding the shape similarity makes easier category ->
encoding the shape similarity enables easier category

similarity makes easier -> similarity enables easier 

p9: embodied agents that interacts -> embodied agents that interact

summary_of_the_review: This is an exciting paper, showing that "natural" temporal interactions with objects significantly boosts the performance of self-supervised algorithms.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes the idea of using time-based data augmentations to aid general purpose machine vision systems. Authors provide a thorough and very detailed psychological motivation for why this may work, and later use simulated agents's view points to simulate the time-based components of learning. Authors find that stacking time-based representations on top of other data-augmentation schemes aids in recognition.
strength_and_weaknesses: See Summary of Review in section below. TLDR: The main paper's strengths are the original scientific question, and methodological execution (dataset, networks and learning regimes used). The main weakness is a tentative missing control to justify the main claim (see summary review section for details).
clarity,_quality,_novelty_and_reproducibility: The paper is easy to read, very clear, and the appendix and level of detail is high enough to guarantee reproducibility. Authors also have states in the paper that they will share their code if the paper is accepted and that is a great idea.
summary_of_the_review: This is a type of paper that everyone talks about that "someone will have to do this", but no one is aware of anyone doing it, so I am happy that a paper regarding training a NN that incorporated the temporal dynamics in a SSL framework is studied and properly evaluated. This paper deserves an accept just because: 1) the authors have tried (and this isn't an obvious thing to do both in terms of finding an interesting scientific question and also the methodology, since most of the CV community is unfortunately still thinking about ImageNet, and developing a convoluted new architecture to increase the validation set performance by 0.1%); 2) authors find a set of interesting results and the cohesive story of the experimental design + evaluations mostly make a lot of senses (however, there's a couple of questions I have later on this).

The main **weakness** I find in this paper is that I think at some point there was some conversation (in the paper) about having a control condition where the video frames were shuffled as a control to eliminate the confounding variable of "more data". Without this control, the paper almost falls short because it still raises the question: **"Do time-based augmentations drive this increase in performance due to their time-based nature, or by pure virtue of adding more training data?"** Maybe authors have added these plots in the Appendix and I've missed this?

Other than that, **this is a fantastic paper(!). I'm willing to increase my score if authors address my concerns in this review.**

* Missing References:

   * On eye-movements and multiple views for robustness, it may be worth adding the recent work of **Harrington & Deza. ICLR 2022** that showed through a set of psychophysical experiments that learning of multi-view foveal and peripheral template potentially aids in robust representations for humans (through adversarial robustness experiments in machines as well).

   * Another key paper that is missing is the theme of time-based contrastive/self-supervised models are: ``Are models trained on temporally-continuous data streams more adversarially robust?'' by **Kong & Norcia. SVRHM 2021.**

Of course, it would be quite interesting to see how time-based learning affects the agents performance in adversarial robustness and/or common corruptions. Perhaps this is not doable to do for the rebuttal (that is ok), but I'd be curious what the authors would think would happen. I expect stronger robustness, but will this actually be the case if some studies have shown that a robustness-accuracy tradeoff exists? (Tsipiras, ICLR 2019)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose to use temporal changes in object views that occur naturally in developing infants' visual environment as data augmentations for time-based self-supervised learning. Using a rendering environment they create positive and negative training pairs using 3d object manipulations such as rotations, saccadic eye movements, moving objects (to change the background). They compare to standard self-supervised learning using data augmentation and evaluate the learned representations on both the rendered and some real-world video datasets of humans manipulation objects.

strength_and_weaknesses: ### Strengths

 + Well motivated overall question
 + Important question how active vision could support visual representation learning beyond the simple data augmentation strategies currently used
 + Using rendering could be an interesting way forward to generate realistic view changes synthetically
 + Implemented an interesting and diverse set of view changes caused by active vision not captured by data augmentation


### Weaknesses

 1. Method does not actually result consistently in improved visual representations
 1. Small size of datasets seems to be a major limiting factor preventing strong conclusions
 1. Motivation for the individual experiments and how they each relate to the main research question is not very clear
 1. Analyses are lacking depth



### 1. No consistent improvement

One major concern is that the results do not really show clear evidence that the proposed approach indeed outperforms standard data augmentations. In more than half of the dataset/learning method combinations it actually performs worse.


### 2. Small datasets

Another major concern is what we can actually learn about self-supervised learning in practically relevant scenarios from such small datasets. The fact that both methods (standard and time-based) combined work significantly better than each one individually suggests that in both cases the number of unique image sources/scenes is a major limiting factor and the combination of both methods simply leads to substantially more "effective" data. How each of the methods would fare in a more typical self-supervised setting where the datasets are large is not really answered by these experiments.


### 3/4. Motivation + lack of depth

The authors present a collection of results, each of which addresses a question that is in principle interesting. However, it was not very clear to me why exactly these questions were asked and what we learn from the analyses. To name just one concrete example, why are saccades only simulated on CORe50 but not VHE? 

Each question is addressed very superficially by one manipulation, often on a single dataset, without discussing robustness and generality of these results: how much do they depend on the particular way the authors operationalized the question, what about different datasets or change in hyperparameters?

Overall, after reading the paper I'm left wondering: what now? What follows for self-supervised learning? The fact that invariance to 3d rotations and changes in background are useful is undoubted, but such changes simply cannot be easily generated from unlabeled images and deriving them from unlabeled video is not a unique contribution of this paper.

clarity,_quality,_novelty_and_reproducibility: The overall motivation and introduction are very clear and compelling. Although several aspects of the work are creative and novel, unfortunately the experiments and results do not really live up to the expectations. 

I expect reproducibility to be limited, since the relatively large number of individual experimental manipulations are not described in much detail, albeit these things could be inferred from the code, which is promised to be published. *[note: in my original review, I had not seen the note about code being published]*

summary_of_the_review: A well-motivated paper with creative and original aspects but relatively poor execution and somewhat unclear conclusions.

### [Update after rebuttal]

Based on the discussions and the other reviewers' impressions I increased my score and gave the paper the benefit of the doubt.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies how augmentations induced by object interaction sessions help with self-supervised learning performance. This paper uses two synthetic environments and two real world data environments, each with different manipulation, ego-motion, and fixation related augmentations for self-supervised learning. The authors showed that these types of augmentations are more helpful than traditional image-space augmentations, and best if combined with those augmentations as well.


strength_and_weaknesses: #### Strength

This paper excels at performing well designed and controlled augmentations to study their performance in various settings. Through this design and careful control, it is convincing to show that the type of augmentation introduced are helpful for SSL.


#### Weaknesses

My main concern for this paper is the notion of "time-based" augmentations. I find this term a bit misleading: most of the augmentation introduced seems time-invariant, but more as 3D augmentations. By time-invariant I mean in the case that if all the frames captured each session are randomly shuffled then piped to the SSL frameworks, augmentations like manipulation and ego motion seems unaffected at all, where all that matters are the set of positive and negative pairs, but not their temporal relations. Next frame positive pair might be affected, which is truly time-based. But according to [1], this may not be a good strategy for mining semantic correspondences. 

I therefore would like to ask the author for more arguments or evidence to support the use of "time", in addition to the 3D-based augmentations provided in the paper. Or a more proper name should be used. 

[1] Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective
clarity,_quality,_novelty_and_reproducibility: - This paper is well written and easy to follow.
- The authors are well thought on the experiments to support their claim, making it a good quality paper
- The augmentations introduced in this paper are not significantly novel perse, but the careful design of the augmentations to mimic human response is likely to be novel.
- If the authors would release their data generation pipeline, this paper is reproducible.

summary_of_the_review: In summary, I find this paper slightly below the bar of acceptance. My major concern is the notion of 'time-based augmentations', where a lot of those augmentations are just 3-D based and are time ignorant. To match this term more closely, the augmentations, or the positive frame selection must be time sensitive. To support the claim that this is helpful, the baseline should be a time-ignorant positive frame sampling strategy with the same set of augmentations. 
Also, the types of augmentations shown here are marginally novel in pure SSL sense, but this could be an underestimate since the augmentations are designed more closely to mimic human responses.

## Update after the rebuttal 
My main concern is mostly around the term 'time-based': some of the augmentations used in this paper is not quintessential in terms of 'time'. One example is rotation of the object, which can be seen as an instance-centric 3D augmentation, where the ordering of different poses along the temporal dimension does not matter. 
The longer phrase "time-based augmentations during natural interactions" is more accurate, and I apprecitate the change proposed by the authors. I do hope the title of the paper can be more accurate as well, but it is a relative minor issue.
Given the author's response, I would like to increase my rating from 5 to 6.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper describes a biologically-motivated approach for time-based data augmentation to be used for self-supervised learning. There was general agreement that the reported improvements over baseline and that the approach was interesting at least from a biological perspective
justification_for_why_not_higher_score: The idea is not completely novel; several authors have proposed similar ideas.
justification_for_why_not_lower_score: There is unanimous agreement that the paper should be accepted.
Decision: Accept: poster