Time to augment self-supervised visual representation learning
Authors: Arthur Aubret
Markus R. Ernst
Céline Teulière
Jochen Triesch
Venue: ICLR 2023 poster (2023)
Keywords: object representations
self-supervised learning
time-based augmentations
data augmentations

Abstract: Biological vision systems are unparalleled in their ability to learn visual representations without supervision. In machine learning, self-supervised learning (SSL) has led to major advances in forming object representations in an unsupervised fashion. Such systems learn representations invariant to augmentation operations over images, like cropping or flipping. In contrast, biological vision systems exploit the temporal structure of the visual experience during natural interactions with objects. This gives access to “augmentations” not commonly used in SSL, like watching the same object from multiple viewpoints or against different backgrounds. Here, we systematically investigate and compare the potential benefits of such time-based augmentations during natural interactions for learning object categories. Our results show that incorporating time-based augmentations achieves large performance gains over state-of-the-art image augmentations. Specifically, our analyses reveal that: 1) 3-D object manipulations drastically improve the learning of object categories; 2) viewing objects against changing backgrounds is important for learning to discard background-related information from the latent representation. Overall, we conclude that time-based augmentations during natural interactions with objects can substantially improve self-supervised learning, narrowing the gap between artificial and biological vision systems.

Published as a conference paper at ICLR 2023
TIME TO AUGMENT SELF-SUPERVISED VISUAL REPRESENTATION LEARNING
Arthur Aubret Markus R. Ernst Céline Teulière' Clermont Auvergne Université, CNRS, Clermont Auvergne INP, Institut Pascal Frankffrt Institute for Advanced Studies (arthur. aubret, celine. teuliere @uca fr (mernst, triesch @faiis.s..............aa uni frankfurt .de
Jochen Triesch
ABSTRACT
Biological vision systems are unparalleled in their ability to learn visual representa- tions without supervision. In machine learning, sef-suuperviee learning (SSL) has led to major advances in forming object representations in unsupervised fashion. Such systems learn representations invariant to augmentation operations over im- ages, like cropping or flipping. In contrast, biological vision systems exploit the temporal structure of the visual experience during natural interactions with objects. This gives access to "augmentations" not commonly used in SSL, like watching the same object from multiple viewpoints or against different backgrounds. Here, we systematically investigate and compare the potential benefits of such time-based augmentations during natural interactions for learning object categories. Our re- sults show that incorporating time-based augmentations achieves large performance gains over state-of-the the-thrrrt art image augmentations. Specifically, our analyses reveal that: 1) 3-D object manipulations drastically improve the learning of object cate- gories; 2) viewing objects against changing backgrounds is important for learning to discard background-relater information from the latent representation. Overall, we conclude that time-based augmentations during natural interactions with objects can substantially improve self-super learning, narrowing the gap between artificial and biological vision systems.
INTRODUCTION
Learning object representations without supervision a grand challenge for artificial vision systems. Recent approaches for visual self-super learning (SSI.) acquire representations invariant to data-augmenttation based on simple image manipulations like crop/resize, blur or color distortion (Grill et al.) 2020; Chen et al.) 2020). The nature of these augmentations determines what information is retained and what information is discarded, and therefore how useful these augmentations are for particular downstream tasks (Jaiswal et al. 202T Tsai et al. 2020). Biological vision systems, in contrast, appear to exploit the temporal structure of visual input during interactions with objects for unsupervised representation learning. According the slowness principle (Wiskott and Sejnowskil
2002, Li and DiCarlo, 2010, Wood and Wood 2018), biological vision systems strive to discard high-frequency variations (e.g. individual pixel intensities) and retain slowly varying information (e.g. object identity) in their representation. Incorporating this idea into SSL approaches has led to recent time-contrastive learning methods that learn to map inputs occurring close in time onto close-by latent representations (Oord et al., 2018 [Schneider et al. 202I). How these systems generalize depends on the temporal structure of their visual input. In particular, visual input arising from embodied interactions with objects may lead to quite different generalizations compared to what is possible with simple image manipulations. For instance, human infants learning about objects interact with them in various ways (Smith et al. 2018). First, infants rotate, bring closer/farther objects while playing with them (Byrge et al.) [014). Second, as they gain mobility, they can move in the environment while holding an object, viewing different contexts and against different backgrounds. We refer to (simulations of) such interactions as natural interactions.
* "qqual contribution.

---

Published as a conference paper at ICLR 2023
Here, we systematically study the impact of such natural interactions representations learnt through cime-contrastive learning in different settings. We introduce two new simulation environments based on the near-pül sear-photorealistic simulation platform ThreeDWorld (TDW) (Gan et al. (2021) and combine them with a recent dataset of thousands of 3ID object models (Toys4k) (Stojanov et al. 2021). Then validate our findings on two video datasets of real human object manipulations, ToyBox (Wang)
it al. 2018) and CORc50 (Lomonaco and Maltoni 2017)
Our experiments show that adding time-based augmentations to conventional Gata-augmentations considerably improves category recognition. Furthermore, we show that the benefit of time-based augmentations during natural interactions has two main origins. First, 3-D object rotations boost generalization across object shapes. Second, viewing objects against different backgrounds while moving with them reduces harmful effects of background clutter. We conclude that exploiting natural interactions via time-contrastive learning greatly improves self-supprvised visual representation learning.
2 RELATED WORK
Data-augmented self-supervised learning. The general idea behind most recent approaches for self-supervised learning is that two semantically close/different inputs should be mapped to close/distant points in the learnt representation space. Applying a certain transformation to an image such as flipping it horizontally generates an image that will be very different at the pixel level, but has a similar semantic meaning. A learning objective for SSL will therefore try to ascertain that the representations of image and its augmented version are close in latent space, while being far from the representations of other unrelated images.
A concrete approach may work as follows: sample image x; apply transformations to it taken from a predefined set of transformations, resulting in new images x', also called a positive pair. The same procedure is applied to batch of different images. Embeddings of positive pairs are brought together while keeping the embeddings of the batch overall distant from one another. There are three main categories of approaches for doing so: contrastive learning methods (Chen et al.) [2020] HH et al.. 2020) explicitly push away the embeddings of a batch of inputs from one another: daiillatioo-asse methods (Grill et al.] 2020) Chen and He| (02T) use an asymmetric embedding architecture, allowing the model to discard the "push away" part; entropy maximization methods (Bardes et al., 2022
Ermolov and Sebe 2020) maintain a high entropy in the embedding space.
Image manipulations data-augmentations. Most self-superviste learning approaches have used augmentations based on simple image manipulations to learn representations. Frequently used are color distortion, cropping/resizing part of an image, the horizontal flipping of an image, gray scaling the image, and blurring the image (Chen et al. (2020). Other augmentations can be categorized in three ways (Shorten and Khoshgoftaar 2019 Jaiswal et al. 202T): 1) geometric augmentations include image rotations (Chen et al.) 2020) or image translations (Shorten and Khoshgoftaar 2019);
2) Context- based augmentations inc include jigsaw puzzle augmentations (Noroozi and Favaro 2016)]
Misra and Maaten 2020), pairing images (Inoue 2018), greyed stochastic/saliency- -based occlusion
(Fong and Vedaldi 2019 Zhong et al.) 2020), or automatically modifying the background (Ryali et al. 2021); 3) Color-oriened transformations can be the selection of color channels (Tian et al.
2020) or Gaussian noise (Chen et al., 2020). A related line work also proposes learning how to generate/select augmentggttnnn (Cubuk et al. 2019 Tian al., 2020 but since it takes advantage of labels, the approach is no longer self-supervised.
Time-based fata-auugmentatios.. Several works have proposed using the temporality of interac- tions to learn visual representations. A recent line of work proposes to learn embeddings of video frames using the temporal contiguity of frames: Knights et al. (021) propose a learning objective that makes codes of adjacent frames within video clip similar, however the system still needs to have information about where each video starts and ends. In contrast, our setups expose the system to a con- tinuous stream of visual inputs. Other methods showed the importance of time-based augmentations based on videos for object tracking (Xu and Wang 2021), category recognition (Gordon et al. 2020 Parthasarathy et al.) 2022 Orhan et al.] 2020) or adversarial robustness (Kong and Norcia 2021).
Unlike us they do not make an in-depth analysis of the impact of different kinds of natural interactions.
Schneider et al. (202)) showed the importance of natural interactions with objects for learning object
2

---

Published as a conference paper at ICLR 2023
Object positions
Agent positions
Figure 1: Top view of the VHE used to situate our agent in house. Blue triangles/orange circles show possible agent/object positions, respectively. In this episode, the agent is located in the office (blue arrow) and interacts with a plush toy. Successive first person views object interactions are shown in the lower left.
representations, but only considered small datasets of few objects, without complex backgrounds and without studying generalization over categories of objects. Wang et al. (202T) have shown that one can replace crop augmentations by saccade-like magnifications between conventional image-based based and more natural time-based augmentations. Aubret et al. (2022) studied the impact embodiment constrains on object representations learnt through time-contrastive learning during natural interactions. They highlight that complex backgrounds negatively impact the representations of objects. Here we show that allowing the agent to move with objects so that they seen against changing backgrounds improves robustness against background complexity.
images, establishing link
We note that a large body of work also considers time-contrastive learning in the context reinforce- ment learning (Oord et al. 2018 Laskin et 22200] Stooke al.) 2021 Okada and Taniguchi 2021)
and intrinsic motivation (Guo et al. 2021 Yarats et al. 2021 et al. 2021 Aubret et al. 2021).
METHODS
Our goal is to study the potential time-based augmentations during natural interactions with objects for visual representation learning and evaluate the utility of the learned representations for object We focus on two kinds of natural interactions: object manipulations (3D translations and rotations) and ego-motion. We introduce two computer-rendent rendered environments, the Virtual Home Environment (VHE) and the 3D Shape Environment (3DShapeE) to compare and disentangle the effects of these different types of natural interactions.
categorization [
We also validate our findings using two real-world first-person video datasets. The ToyBox environ- ment (Wang al. (2018) permits studying the relative benefits object translations and rotations. The CORe50 environment (Lomonaco and Maltoni (017)) allows us explore the effects of smooth visual tracking and to mimic the effects ego-motion.
of
3.1 VIRTUAL HOME ENVIRONMENT
To simulate diverse set of natural interactions of infant in its home, we take inspiration from (Aubret et al. 2022) and place an agent within an environment that resembles a residential house, where can "play" with objects and change positions. Figure deppicss bird's eye view showing the floor plan of the afformmentioned house. In comparison with Aubret et al. (2022). ) we integrate a two order of magnitude higher number of fully textured toy objects, well novel interactions with these objects. The agent is spawned in randomly sampled location, staying there for 100-timestep-long
'The code is available at http://ghhiiiccommiicce////tttte

---

Published as a conference paper at ICLR 2023
B
Figure 2: Examples of time-based augmentations while "playing" in the VHE (left, A-C): A) object rotation; B) ego-motion; C) depth change. CORe50 (right, D-E): D) close-by frames of objects in 2 different recording sessions; E) different exemplars of the object category "cellphones" shown in different contexts. ToyBox (left, F-H): F) depth change, G) lateral translation, H) object rotation.
episodes (or sessions). Its position in the world determines what background will be seen. Each background contains a unique combination of static objects and floor/wall textures.
Toys4k dataset. The agent interacts with objects from the Toys4k dataset, which is composed of 4,179 diverse 3-D toy objects distributed in an unbalanced way into 105 categories (Stojanov 202Tp. was designed to match the objects typically encountered by infants. We place textured objects in different locations of the full house. We modify/remove 3D models, only keeping baick-to-process and correct cerrect-quality objects in our simulations (ef. Appendix [. more details). To study the ability of our agent to generalize over categories, we use two thirds of the objects of each category for training and keep the last third for testing. We construct a test set composed of randomly rotated views of the test objects in different locations of the house. Except for rotation, no additional transformations have been applied to build the test set. We also introduce novel houses to test the robustness of the object-representation with respect to novel -utto--distribution backgrounds (cf. Appendix C.2).
Object manipulations. At only a few months old, infants start hold objects and move and rotate them (Byrge et al., 2014). Similarly, our agent can manipulate the objects in two different ways. First, can rotate the object around the vertical axis (yaw) by random angle drawn uniformly from the interval [o. rot] degrees, where the hyperparameter rot sets the maximal speed of object rotations (cf. Figure A)). Second, the agent can also bring the object closer or move it further away, as shown in Figure 2C; we model this as change of distance uniformly sampled from [-d, dJ. where 7.5 cm is the maximal distance change. The distance is bounded between 0.65 m and 1.1 m to reflect morphological limitations of the agent's 's arms.
Ego-motion. When infants become mobile, they can play with objects while moving in their environment. To simulate this, our agent can turn its body towards a neighboring location while holding the same object so that it is seen against different background. As shown in Figure 2B, the new background will usually have similar floors and walls. Sometimes an infant will also engage with new object. We model both possibilities by deterministically turning the agent every N, time steps and switching to a new object with probability - P. 0.1. small/medium N₂ respectively simulates in unnatural/natural fashion agent that always/sometines moves with its object. When this ego-mon ago-motion is disabled, the agent simultancously picks up a new object and turns every NN = 10 time steps. For some experiments (Table I and Figure 5J. B). we also allow the agent to visit another room every N, steps while continuing to interact with the same object. This allows us to investigate the effect of more drastic background changes, but may be less valid model of human infant's interaction with objects.
3.2 3D SHAPE ENVIRONMENT
In order to study the influence natural interactions on shape generalization while minimizing the impact of object colors or background, we introduce simple 3D Shape Environment (3DShapeE). It differs from the VHE in two respects: 1) we import untextured versions of toy objects from the Toys4k dataset, making them appear grey; 2) place the agent in empty environment so that objects seen against a blank background. Examples of inputs are given in Figure and augmentations are the same as in the VHE environment (Figure (FigureAA-C) A-C).
4

---

Published as a conference paper at ICLR 2023
3.3 ToYBoX ENVIRONMENT
The BoyBox dataset (Wang et al. 2018) contains first-person videos of observer manipulating 360 objects (12 categories, 30 objects each). In each clip, a person applies one type of transformation: translation (x, y or z), rotation (around x, y or Z axis), a "hodgepodge." nothing specific, or some cases, nothing if there is no object. We sample two frames per second from each clip (cf. Appendix C3 for more details). Two successively sampled frames form positive pair, At the end of a clip, positive pair is formed from the last frame sampled from the clip and the first image of the randomly sampled next clip.
3.4 CORE50 ENVIRONMENT
The CORe50 dataset (Lomonaco and Maltoni 2017) contains first-person views of observer manipulating objects. The dataset comprises 167,866 images of 50 different objects belonging to 10 distinct object categories. All objects were filmed in 11 sessions corresponding to different natural environments. The images from all but one object per class are used to form the training and validation sets. Specifically, every 10th image enters the validation set, the others form the training set. The images of the held-out object of each class form the test set. This allows us to test generalization to unknown objects from familiar categories.
The videos in this dataset show real object manipulations, but we also use it to mimic the effects of ego-motion through the way we sample images from the different video clips. In general, successively sampled images form positive pairs. We create sequences of images by starting from particular frame of one of the video clips showing an object manipulation and defining a probability Po that the manipulation of the same object continues. Otherwise an interaction with new object will start. In case the interaction continues, the next frame in our sampled sequence will be randomly chosen as either the next or the previous frame in the video clip. Thus, as long the object manipulation continues, the sampled sequence of frames corresponds to a random walk through the frames of the video. A high/low Po leads to long/short random walks. Specifically, the expected number of successive frames No showing the same object is given by:
DC
E [Nopp. = 1+1 Σ
2y (1 - po) = 1 Po
1 Po
To mimic the effects of ego-motion causing an object to be seen against changing backgrounds, we also define a probability Pa that the sequence of views stays within the same session, i.c., the same video clip. Thus, there is a chance of - Ps that the sequence of views of the same object "jumps" to different video clip showing the same object in different context. In this case, positive pair is generated from two images of the same object recorded in different recording sessions. This cross session sampling (CSS) is analogous to the role of ego-motion in the VHE.
As an alternative to the random walk procedure described above we also consider a variant where the next view of the same object is chosen uniformly at random from the current video clip. This leads to images forming a positive pair, which stem from distant time points in the original video clip. While this is not realistic model of viewing sequences experienced by an infant, it allows us to assess the benefits of positive pairs containing more distinct views of the same object. This is analogous to greater speed of object rotation the VHE.
For implementation details of our sampling procedure see Appendix B.5 In brief, it is an extension of the procedure used in SSchneider et al. (202I) to allow CSS.
3.5 EVALUATION AND TRAINING PROCEDURE FOR ALL ENVIRONMENTS
Learning algorithms. We consider three SOTA representatives of existing algorithms for data- augmented SSL: SimCLR for contrastive learning (Chen et al. 2020), B BYOL (Grill et al. 2020) for distillation-bar based methods and VICReg (Bardes et al. (022)) for entropy maximization methods. We refer to the versions of these algorithms using time-based augmentations as, e.g., SimCLR through time (SimCIR-TTT) etc. (Schneider al. 2021
5

---

Published as a conference paper at ICLR 2023
Time
Source images
--TT
Baseline
+-TT+
Figure 3: Sequences of input images during object rotations (rot(360)) and ego-motion and their augmented positive pair according to the augmentation method (same column). Baseline refers to conventional augmentations; *-TT refers to time-based augmentations; *-TT+ refers combined augmentations.
Positive pair construction. We consider three ways to augment image. First, consider time-based augmentations during natural interactions (SimCLR-TT etc.). For this, unless stated otherwise, we the positive pair of a source image the next image in the temporal sequence. Second, as comparison baseline, we augment a sampled image with widely adopted SOTA set of augmentations (Chen et al. 2020 Shijie al.) 2017). i.e. crop/resize, grayscale, color jittering and horizontal flip with their default parameters, except for the crop/resize augmentation (ef. Appendix D). Third, we propose to combine the two kinds of augmentations (SimCI--TT4+ etc.). For this we apply the conventional SOTA augmentations to the image that temporally follows the sampled source image. We refer to Figure for examples of the different ways to augment source images.
Controlling for the number of augmentations. While we can generate an arbitrary number of positive pairs from one sample image with conventional augmentations, we only have access to one positive time-based pair per sample image for the VHE, 3DShapE, and the ToyBox environment. To control for the impact of the number of positive pairs, we therefore only construct one positive pair per sample for all algorithms and augmentation methods: for the ToyBox environment, we sample positive pairs from pre-built dataset of the same size as the original one, but composed of conventionally augmented images; 2) for the VHE and 3DShapeE the agent collects images, augments them and stores them in buffer of size 100,000. It simultaneously trains images randomly sampled from the buffer at every time step. We show in Appendix B.I that controlling for the number of augmentations does not impact the results of our analysis.
Controlling for the number/distribation of source images. We want to make sure that we compare the quality of the augmentations, and not the amount/diversity of data that feeds the SSL algorithms. Thus, for the comparison baseline, we consider the same natural interactions generate source images, but we do not use temporal information to generate the positive pairs. This ensures that the number/distribution of non-augmented images is exactly the same across methods (source images in Figure 3). We make an exception with depth changes in 3DShapeE since we found to be harmful for the baselines.
Evaluation. We compute the representation with ResNet18 on ToyBox and CORe50 and with simpler convolutional neural network on 3DShapeR and VHE. We evaluate the learnt representation after 480,000 steps for the VHE and 3DShapeE, 60 epochs for ToyBox and 100 epochs for CORe50. The quality of the learned representation is assessed by training linear readouts on top of the learned representation in a supervised fashion (Chen et al. 2020). Depending
the question, we train linear readouts to either predict the identity of an object, the category of an object, or the location/session of the object. We average all results over random seeds. Because of the small number of objects per category in the CORe50 dataset, we apply a cross- cross-validttion with 5 splits. Hyperparameters are given in Appendix [

---

Published as a conference paper at ICLR 2023
Method
VHE
3DShapeB
ToyBox
CORe50
SimCLR
0.328 + 0.002
0.527 1 0.005
0.313 = 0.001
0.544 1: 0.074
SimCLR-TT TT
0.369 = 0.005
0.598 + 0.007
0.189
0.009
0.449 0.152
SimCLR-TT+
0.537 + 0.001
0.621 + 0.001
0.378 = 0.005
0.610 + 0.090
BYOL
0.352 = 0.017
0.496 + 0.012
0.354 = 0.007
0.552 0.063
BYOL-TT
0.369 0.007
0.570 + 0.001
0.213 = 0.014
0.448 + 0.160
BYOL-TT+
0.516 0.002
0.536 + 0.001
0.393 = 0.031
0.614 + 0.128
VICReg
0.169 0.006
0.512 0.002
0.381 = 0.019
0.410 + 0.073
VICReg-TT
0.264 = 0.012
0.383 + 0.167
0.158 = 0.008
0.245 + 0.084
VICReg-TT+
0.435 + 0.013
0.574 + 0.008
0.418 = 0.016
0.503 + 0.057
Table 1:
Top accuracy + standard deviation (over seeds, CORe50 over training/testing splits) under linear evaluation of previous SOTA (BYOL, SimCLR, VICReg) versus time-based augmen- tations (*-TT) and their combination (*-TT+). In 3DShapeE. +-TT+ use crop/resize, color jittering and rotations. In VHE, *-TT+ use rotations, N₂ with room changes, color jittering and gray scaling. In both VHE and 3D) Shape Environments, *.TT use rotations, N₂ = with room changes (VHE only) and depth changes. In CORe50 for the ablation study that motivates these choices. In ToyBox, we always apply all their respective transformations. Unlike * *-TT+, we apply all standard augmentations for the baseline, as we found that removing some of them (crop/resize and flip) was harmful.
p3, 0.5, Po 0.9. We refer to Appendix [B.2
4 RESULTS
In this section, we first study the benefits of using time-based augmentations during natural interac- tions with objects for self-supervised visual representation learning. Then, we provide an in-depth analysis of the impact of 3-D object rotations and ego-motion the learned representations. An ablation study is performed in Appendix B
4.1 TIME-BASED AUGMENTATIONS DERIVED FROM NATURAL INTERACTIONS BOOST PERFORMANCE OF SSL
In Table [ we compare the quality of the learnt representation when using conventional data- augmentations (SimCLR, BYOL, VICReg), time-based augmentations (*-TT), and their combination (*-TT+) in our four test environments. We observe that our combined approach (*-TT+) increases the average test category accuracy with respect the the baselines (SimCLR, BOL, VICReg) by range of [0.037; 0. 0.266] points across all environments and all SSL methods. To formally validate approach, we additional experiments for SimCLR and SimCLR-TT- and applied t-test to statistically compare their performance, supporting the superiority SimCLR-TTE (p < 0.05 in our four test environments). Time-based augmentations during natural interactions (*-TT) do perform very well on their own, presumably because they do not create invariance to color/grayscale changes in our test environments (cf. Appendix B for ablation studies of augmentations). In the sections, we analyse the impact of 3-D rotations and ego-motion on the learnt representations.
4.2 ОВJECT MANIPULATIONS INCLUDING 3-D ROTATIONS SUPPORT SHAPE GENERALIZATION
To assess the importance of object manipulations to generalize over object shapes, we conducted experiments in 3DShapeE. Figure 4A shows that increasing the speed of object rotations systemati- cally increases the category recognition accuracy. Similarly, in the CORe50 Environment (Figure ( the uniform sampling of views clearly outperforms the default random walk sampling procedure, in particular for short object manipulations (low Po). Interestingly, we observe the inverse effect on individual object recognition accuracy (Figure 4B). We conclude that fast object rotations (in the extreme constructing positive pairs from randomly rotated views of an object) tend to discard object-specific details and focus the representation on more general shape features, which will be similar for different objects of the same category.
7

---

Published as a conference paper at ICLR 2023
calogery
30ShapuE catogery
CCRu55 bext cellgory scc.
Fiortslers
(0P (0),40)
07
0.5
8.
0.
0.8
0.0
0.P
63
0.0
0.1
0.2
0.3
78
E
Figure 4: A) 3DShapeE test category recognition accuracy for different maximal rotation speeds S, without other augmentations. B) 3DShapeE test object recognition accuracy for different maximum rotation speeds, without other augmentations. The shaded area displays the +/- standard deviation of test category accuracy over seeds. C) Test category accuracy with a linear classifier trained on one single orientation of objects, and tested the same orientation of test objects. We used 5 seeds, all our conventional augmentations but only rotations ((0, 360] degrees) as time-based augmentation. We also apply the majority vote on datasets of six different orientations: 0", 10", -45%, 80 -800, -90". D) Category accuracy for the CORe50 Environment when varying the duration of object manipulations (p& 0.95). The shaded area shows standard error. A-D) We used SimCLR-TV E) Example of three views for eight objects from four categories in 3DShapeE. In order from left to right for each object, we show orientations of: 0% (side view); -450 (34 view); 90º (back view). The red rectangles highlight visually similar views of different objects.
Finally, we investigate the relation between rotation invariance and object shape. To do so, we design new test setup. In the single-single setup, we train the linear classifier the representations of single-view dataset that contains only one view from a common orientation for each training object (back, ...). We assess the classifier images of the test objects shown in the same orientation. This means that the linear classifier can not learn the invariance to rotation and does not explicitly need it for classification. We consider datasets of orientations of -90" (front view), -450 (34 view), 0" (side view) as they contain the extreme cases in terms of the amount of visible surface (Blanz et al.. 1999).
We hypothesize that including 3-D rotations in time-based augmentations allows the model to encode 3-D shape features in the learned representation. which results an improved category recognition. To test this hypothesis, we first evaluate category recognition based on several views of an object (thus its whole shape). In Figure we compute a majority vote among classification results from the different single-single classifiers as follows: if more views (e.g., 90", 80 10") are classified selected. We observe a large increase of accuracy for SimCLR. We conclude that encoding the 3-D shape improves category recognition. Importantly, we do not observe such an increase of accuracy for SimCLR-TT with the majority vote. We deduce that SimCLR-TTI better infers the 3-D shape of the object from one view, which strongly improves category recognition.
as category A than category B (e.g., 0", -90"), category
4.3 MOVING WITH OBJECTS MITIGATES THE EFFECT OF BACKGROUND CLUTTER
To evaluate the impact of ego-motion while holding objects, we trained our agent in the VHE with different levels of motion. In Figure we see that increasing the frequency I/N, and the range (room) of motion considerably boosts the test category accuracy in backgrounds from training and novel houses. At the same time, we observe decrease of the accuracy when classifying the background (Figure 5B.. We conclude that moving while holding objects improves the representation 's invariance with respect to the background.
Moreover, we analysed the frequency of session changes compared to the duration of object manipu- lations one of the hyperparameters of our approach in the CORe50 Environment, see 5C.
8

---

Published as a conference paper at ICLR 2023
A
VHE culogory
B
VHE badiground
Teat calleory ado
Test session 80c
1,20
1,00
704
0.4
0.75
0.75
Ao0a000
10
S(N
.10 715 752
0:65
22.00
0.3
0.60
20
0.25
0.25
A40
AM ARS
704. 207
.75
751
0.06
0.2
5
10
20
50
10
20
50
Trail
E[N.1
Figure 5: and Impact of the frequency (Na) of body motions and whether they change rooms on the final test category accuracy (A) and the final background accuracy (B). We also used object rotations ((0, 360] degrees). color jittering and gray scaling. Vertical bars indicate one standard deviation. C and D. Mean CORe50 accuracy for category (C) and session (D) classification dependent on the duration of object manipulations (N,) and the frequency of session switches (N,). SimCLR- TT, random walk procedure, mean based on training/testing splits.
D. Qualitatively, for category classification we observe low accuracies for high-frequency object switches, low session changes and high accuracies for low-frequency object switches, high session changes. Accuracy seems more or less saturate at the main diagonal. We infer from this that inter- mediate values may actually be better than going to the extreme, but it could also partly because N₂ is bounded by No- The classification of sessions consistently shows that session information is being disregarded when the frequency of session changes rises.
5 CONCLUSION
We investigated the benefits time-based aataanaugmenttioos during infant-inspired natural object interactions for self-supervised visual representation learning. We studied these interactions through a novel 3-D simulation environment where an agent "plays" with objects in a house, and we validated findings two real-world video datasets of object manipulations. We find that combining time- based augmentations with conventional data-augmentattons (*-TT+ algorithms) greatly improves the ability of the learned representations generalize over object categories. Our analysis shows that 1) 3-D object rotations are crucial to build good representations of shape categories. 2) Ego-motion while holding an object so that it is seen against different backgrounds prevents the background from cluttering the learned representation.
Our work raises the question whether time-based augmentations during natural interactions with objects could fully replace conventional ones. Indeed, we showed that 3-D rotations are superior to the flip data-auge -al---aammmtttionn for shape generalization (Appendix [B.3). However, we found that color-based transformations like color jittering and grayscaling remain important. It is an interesting question for future work time-based augmentations during changes of direct and indirect lighting or shadows cast by other objects could replace augmentations based on simple image color manipulations.
Our work has number of limitations. First, while we have portrayed the time-based augmentations as stemming from object manipulations, we did not render a hand holding the object or similar in the VHE and 3DShapeF. Second, the CORe50 and ToyBox environments are only medium-sized. Large scale experiments may give additional insights. Finally, in this work, only considered naive behavioural strategies for, e.g., turning objects or ego-motion. This contrasts with the way infants learn about their environment (Bambach al.) 2016). For example, they biased towards creating planar views of objects and often prefer unfamiliar objects (Roder et al.) (000). Thus, we expect that learning actively select successive views will further unveil the potential of time-based augmentations.
ACKNOWLEDGEMENT
This work was sponsored by a public grant overseen by the French National Agency through the IMobS3 Laboratory of Excellence (ANR-16-ID) (6-IDEX-000)), Financial support was also received from Clermont Auvergne Metropole through French Tech-Clermont Auvergne professorship. We gratefully acknowledge support
and the IDEX-ISITE initiative CAP 20-25

---

Published as a conference paper at ICLR 2023
from GENCI-IDRIS (Grant resources needed for this work. Additional support was received by the Deutsche Forschungsge- meinschaft (DFG project "Abstract REpresentations in Neural Architectures (ARENA)"). well as the projects "The Adaptive Mind" and "The Third Wave of Artificial Intelligence" funded by the Excellence Program of the Hessian Ministry of Higher Education, Science, Research and Art (HMWK). JT was supported by the Johanna Quandt foundation.
for providing computing and data processing
REFERENCES
Aubret, A., Hassas, S., et al. (2021). Distop: Discovering a topological representation to learn diverse and rewarding skills. arXiv preprint arXiv:2106.0385.
Aubret, A., Teulière, C., and Triesch, J. (2022). 'Toddler-inppired embodied vision for learning object representations. IEEE 21st International Conference on Development and Learning (ICDL).
Bambach, S., Crandall, D. J., Smith, B., and Yu, C. (2016). Active viewing in toddlers facili- tates visual object learning: An egocentric vision approach. In Proceedings of the 38th Annual Conference of the Cognitive Science Society, pages 1631-1636. Cognitive Science Society.
Bardes, A., Ponce, J.. and Lecun, Y. (2022). VICReg: Variance-invariance--ovva for self-supervised learning. Proceedings of the 10th International Conference on Learning Representations (ICLR).
regularization
Blanz, V., Tarr, M. J., and Bülthoff, H. H. (1999). What object attributes determine canonical views?
Perception, 2865)-555-59.
Byrge, L., Sporns, O., and Smith, L. B. (2014). Developmental process emerges from extended brain-body- networks. Trends Cognitive Sciences, 18(8):395-403.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Chen, X. and He, K. (2021). Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750-15758.
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. (2019). Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR).
Ermolov, A. and Sebe, N. (2020). Latent world models for intrinsically motivated exploration. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 5565-5577 Curran Associates, Inc.
Fong, R. and Vedaldi, A. (2019). Occlusions for effective data augmentation in image classification. In 2019 JEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 4158 4166.
Gan, C., Schwartz, J., Alter, S., Mrowca, D., Schrimpf, M., Traer, J., De Freitas, J., Kubilius, J., Bhandwaldar, A., Haber, N., et al. (2021). Threedworld: A platform for interactive multi-modal physical simulation. Thirty-ifihh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round l).
Gordon, D., Ehsani, K., Fox, D., and Farhadi, A. (2020). Watching the world go by: Representation learning from unlabeled videos. arXiv preprint arXiv:2003.07990
Grill, J.-B., Strub, F,, Altché, F., Tallec, C., Richemond. P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., and Valko, M. (2020). Bootstrap your own latent
approach to self-supervised learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 21271-21288 Curran Associates, Inc.
10

---

Published as a conference paper at ICLR 2023
Guo, Z. D., Azar, M. G., Saade, A., Thakoor, S., Piot, B., Pires, B. A., Valko, M., Mesnard, T.,, Latti- more, T., and Munos, R. (2021). Geometric entropic exploration. arXiv preprint arXiv:2101.0205:
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Inoue, H. (2018). Data augmentation by pairing samples for images classification. arXiv preprint arXiv:1801.0290 180..999...
Jaiswal, A., Babu, A. R., Zadeh, M. Banerjee, D., and Makedon, F. (2021). A survey contrastive self-supervised learning. Technologies, 9(1).
Knights, J., Harwood, B., Ward, D., Vanderkop, A., Mackenzie-Ross O., and Moghadam, P. (2021). Temporally coherent embeddings for self-supervised video representation learning. In 2020 25th International Conference Pattern Recognition (ICPR), pages 8914-8921. IEEE.
Kong, N. and Norcia, A. (2021). Are models trained on temporally-continuous data streams more adversarially robust? In SVRHM 202/ Workshop @ Newr!PS.
Laskin, M., Srinivas, A., and Abbeel, P. (2020). Curl:
Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR.
Li, N. and DiCarlo, J. J. (2010). Unsupervised natural visual experience rapidly reshapes size-invariant object representation in inferior temporal cortex. Neuron, 676))::062-1075.
Li, S., Zheng, L., Wang, J., and Zhang, C. (2021). Learning subgoal representations with slow dynamics. In International Conference on Learning Representations (ICLR).
Lomonaco, V. and Maltoni, D. (2017). Core50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, pages 17-26. PMLR.
Loshchilov, L. and Hutter, (2018). Decoupled weight decay regularization. International
Conference Learning Representations (ICLR).
Misra, I. and Maaten, L. v. d. (2020). Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR).
Noroozi, M. and Favaro, P. (2016). Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference Computer Vision (ECCV), pages 69 84. Springer International Publishing.
Okada, M. and Taniguchi, T. (2021). Dreaming: Model-based reinforcement learning by latent imagination without reconstruction. In 2021 IEEE International Conference on Roboties and Automation (ICRA), pages 4209 4215. IEEE.
Oord, A. v. d.. Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint 1xX7v01807.0377
Orhan, E., Gupta, V., and Lake, B. M. (2020). Self-supervised learning through the eyes of a child. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances Neural Information Processing Systems, volume 33, pages 9960-9977 Curran Associates, Inc.
Parthasarathy, N., Eslami, S., Carreira, J., and Hénaff, O. J. (2022). Self-supervised video pretraining yields strong image representations. arXiv preprint arXiv:2210.000
Roder, B. J., Bushnell, E. W., and Sasseville, A. M. (2000). Infants' preferences for familiarity and novelty during the course of visual processing. Infancy, 144):991-507.
11

---

Published as a conference paper at ICLR 2023
Ryali, C., Schwab, D. J., and Morcos, A. S. (2021). Learning background invariance improves generalization and robustness in self-supervised learning on imagenet and beyond. In NeurlPS 2021 Workshop on ImageNet: Past, Present, and Future.
Schneider. F., Xu, X., Ernst, M. R., Yu, Z., and Triesch, J. (2021). Contrastive learning through time. In SVRHM 2021 Workshop @ NeurlPS.
Shijie, J., Ping, W., Peiyi, J., and Siping, H. (2017). Research on data augmentation for image classification based on convolution neural networks. In 2017 Chinese automation congress (CAC), pages 4165-4170. IEEE.
Shorten, C. and Khoshgoftaar, T. M. (2019). A survey image data augmentation for deep learning. Journal of Big Data, 6(1):60.
Smith, L. B.. Jayaraman, S., Clerkin, E., and Yu, C. (2018). The developing infant creates a curriculum for statistical learning. Trends in Cognitive Sciences, 22(4):325-332
Stojanov, S., Thai, A., and Rehg. J. M. (2021). Using shape to categorize: Low-shot learning with an explicit shape bias. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1798-1808.
Stooke, A., Lee, K., Abbeel, P., and Laskin, M. (2021). Decoupling representation learning from reinforcement learning. In Meila, M. and Zhang, T.. editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9870-9879. PMLR.
Tian, Y. Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P. (2020). What makes for good views for contrastive learning? In Larochelle, H., Ranzato, M., Hadsell, R.. Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 6827-6839. Curran Associates, Inc.
Tsai, Y.H.. H., Wu, Y., Salakhutdinov, R., and Morency, L.-P. (2020). Self-supervised learning from a mulli--ieww/a perspective. In International Conference on Learning Representations (ICLR).
Wang, B., Mayo, D., Deza, A., Barbu, A., and Conwell, C. (2021). On the use of cortical magnification and saccades biological proxies for data augmentation. In SVRHM 2021 Workshop@ NeurIPS.
Wang, X., Ma, T., Ainooson, J., Cha, S., Wang, X., Molla, A., and Kunda, M. (2018). The toybox dataset of egocentric visual object transformations. arXiv preprint arXiv: /806.0880000000 06034.
Wiskott, L. and Sejnowski. T. J. (2002). Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4):715-770,
Wood, J. N. and Wood, S. M. (2018). The development of invariant object recognition requires visual experience with temporally smooth objects. Cognitive Science, 42(4): 1391-1469
Xu, and Wang, X. (2021). Rethinking self supervied correspondence learning: video frame-level similarity perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10075-10085.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. (2021). Reinforcement learning with prototypical representations. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1 11920-1 1931. PMLR.
Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Y. (2020). Random erasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
12

---

Published as a conference paper at ICLR 2023
A SAMPLING VIEWS IN THE CORE50 ENVIRONMENT
To simulate No and N₂ we do not directly sample from the data set, but rather build a buffer of positive pairs and iterate through it.
Building the training buffer done in cycles, see Figure 6A. In every cycle, each different object of the data set is chosen exactly once the order in which the objects are presented is new random permutation during each cycle. For each sampled object, the procedure consists of three steps:
1. We randomly sample a view 01-
2. We sample a view from the same object 1/2 to form the same-object pair (v1, u2) using a) random walk, b) uniform. If we reach N,. we sample from different session (same object but with a different background).
3. We repeat 2. form (U2, vz) and continue until reached N, of object manipulation steps.
After that, we sample view from the next object according to the cycle (which also positive pair) and repeat the procedure. Thus, a cycle consists of total of Nobj No object views. During the manipulation of one object, multiple cross-session transitions can occur, see Figure 6B. Several cycles of Nobj X No views are stored in the buffer after another.
During training, batches of pairs of subsequent views are sampled uniformly from the buffer (dashed brackets) and filled into the batch. We consider an epoch to be an entire run through the buffer, which is chosen to match the size of the underlying CORe50 dataset. In consequence, every training image on average is presented once each epoch.
A randomwalk, No =
randomwalk, =o N==
Figure 6: Dynamic sampling procedure for the CORe50 dataset. A) Buffer with deterministic
No = 3. Digits correspond to frame number of the video. B) Buffer with cross session sampling, colors represent different sessions.
B COMPLEMENTARY ANALYSIS
B. ADDITIONAL CONTROL EXPERIMENTS
The number of conventional positive pairs per sample does not matter. To fairly compare the quality of time-based augmentations with conventional ones Table I we controlled for the number of conventional positive pairs in VHE, 3DShapelE and ToyBox by fixing positive pair per image. Here, we aim to check whether time-based augmentations are useful even we increase the number of positive pairs per sample new one for each training minibatch). In this case, the number of positive pairs per sample equals the number of epochs.
In the ToyBox environment (Figure 7A), we observe increasing the number of conventional positive pairs improves both the baseline (SimCLR) and the combined augmentations (SimCLR-TVV and does not change their relative ordering. addition. does not significantly impact the accuracy the 3DShapeE (Figure B) and VHE (Figure Overall, this confirms that time- based augmentations provide complementary and better information about the object shape in comparison to conventional data-aaggeenatiioss.
The linear classifier learns about rotation invariance, but it does not fully explain our results. Here, want assess the importance of the role of linear classifiers to learn the invariance over rotations in our results. We focus on the VHE and reuse the single-single setup used Section and compare the results with new multi-single setup. Through the multi-single setup, train
13

---

Published as a conference paper at ICLR 2023
A
Toyeox category acc.
3DShapeE category acc.
C
VHE category acc.
0.4
SimOLR-TT+ SimCLR SimOLR-TT
0.3
Accurity
Accuracy
0.4
Accuracy
0.2
0.1
0.0
0.0
0.0
sug
Bug Used augrenttations
aug
aug
Used sugmentations
Used augmentations
Figure 7: Test category accuracy according to whether we use one positive pair per image (One aug) several positive pairs, one per minibatch (More aug). We compare on A) the ToyBox environment; B) 3DShapeE and C) VHE.
A
3DShapaE catagory acc, multi-aingla
B
30Shhppe categary acc, singlo-singla
Front views Side views
Frant views
Srde views
0.6
Mayority
0.8
views
Acaussy
Acaurac
0.0
0.5
0.4
0.4
SMOLL-T
S/MCLR
SIMCLR-T
S.CLR
Method
Method
Figure 8: A) Test category accuracy with a linear classifier trained on the multi-view dataset (several orientations objects), and tested on test objects with one orientation. B) Replicated from Figure for readability. Test category accuracy with linear classifier trained on one single orientation of objects, and tested on the same orientation of test objects. We used seeds, all our conventional augmentations but only rotations ((0, 360] degrees) time-based augmentation. We apply the majority vote on datasets of six different orientations: 0%, 10 45%, 80", - -90%.
a linear classifier on representations gained from the usual multi-view dataset, but test it on the single-view datasets.
We observe an overall decrease performance in SimCLR when testing with the multi minti-single setup (Figure (A)) in comparison single-single setup (Figure BB.. In contrast, SimCLR-TTE (with rotations only) more robust to the training set of the linear classifier. It suggests that learning rotation invariance with the linear classifier is mostly harmful for SimCLR. However, the remaining performance discrepancy in the single-single setup shows that it does not explain the entire performance discrepancy in Table
TT+ significantly outperforms its SimCLR counterpart. As mentioned in the main text underline the results presented in Scction[4.]]by statistically testing our SimCLR-TT+ method against its SimCLR counterpart for all of our 4 different datasets/enviromeents We did so by rerunning the experiment with additional random-seeds and comparing test-set accuracy with a two sample independent t-test. We found significant differences for all of our comparisons p .05, the details of this analysis can be found in Table
SimCLR-TT+ also improves over SimCLR in VHE and 3DShapeE with a ResNet18. We wanted to verify whether 77+ during object rotations improves over 7T VHE and 3DShapeE with ResNet18 neural network. In Table 3] repeat the SimCLR-based experiments described in Section 4 with the following modifications: we use a ResNet18 backbone; 2- we set the object manipulation time to 50 time steps; 3- increase the number of positive pairs per sample (like
14

---

Published as a conference paper at ICLR 2023
SimCLR
SimcCL--T++
M
SD
M
SD
df
t(df)
VIEE
0.326
0.004
0.534
0.004
18
112.22
2.2
10 16
3DShapeE
0.523
0.004
0.617
0.002
17
55.477
2.2
10
ToyBox 0.315
0.004
0.378
0.007
18.409
4.8
10 10
CORe50
0.554
0.070
0.629
0.081
18
-2228
0.039
Table Detailed results of the statistical comparison between SimCLR and SimCLR-TT+ regarding test-set accuracy. P-values are bold.
Environment
SimCLR
SimCLR-TT
SimCLR-TT+
3DShapeR
0.617 + 0.007
0.598 + 0.001
0.674 + 0.001
VHE
0.626 + 0.002
0.43 + 0.007
0.634 + 0.003
Table 3: Results of different augmentations with SimCLR R and ResNetl 8 on the VHE and 3DShapeF: environments.
we change the maximum rotation speed in the VHE from 360 degrees to 10 degrees, as we found it to work better in this case. We hypothesize that high rotation speeds favor the encoding of color information, since we did not observe this effect 3DShapeE. We observe that SimCLR-TT+ gives the best performances in both environments.
Appendix B.I :
B.2 ABLATION OF DATA-AUGMENTATIONS
In this section, do ablation experiment of data-an augmentations in ToyBox, 3DShapeE and VHE since we can choose to remove positive pairs that come from specific natural interactions. We also consider conventional data-augmentations in VHE and 3DShapeE since found the complete of augmentations to be suboptimal.
Shape Environment. In Figure D we test different sets of augmentations in the 3DShapeE 3D, progressively and cumulatively combined. Since the combinatorial combination of all augmentations is computationally infeasible, we run greedy forward algorithm to select the order of augmentations:
A
Test category
B
Test category
0.62
0.60
Average accuracy
Average accuracy
0.61
0.65
0.60
0.50
0.59
d-0.075
d=0.15
d=0.3
0.58
0.45
d=0.45
no depth
R
RC
RCJ
RCJD ROJG
RCJH
RD
0.0
0.1
0.4
0.5
Used sugmentations
Million steps
Figure 9: A) Ablation analysis of all data-auummettaions in the 3D Shape environment. Horizontal bars indicate the minimum and maximum accuracy over 3 seeds. We note: R=Rotations; D=Depth changes; J=Color Jittering: G=Grayscale; C=Crop and Resize; H-Horizontal flip. B) Analysis of the impact the maximal change depth distance d.
15

---

Published as a conference paper at ICLR 2023
B
VHE category
0.4
ToyBox category acc.
0.5
0.3
accuray
0.4
0.1
REJGD
All
-Ho -Ho, -H,,
-T
augmentations
Used sugmentations
Figure 10: A) Ablation analysis of all data autaaamummatiinns the VHE. Horizontal bars indicate the minimum and maximum accuracy over 3 seeds. We note: R=Rotations; D=Depth changes; J=Color littering; G=Grayscale; C=Crop and Resize; H=Horizontal flip; E= with Ng=1 and room changes. Ablation study on the ToyBox environment. We apply SimCLR-TT4 but remove hodgepodge (-Ho), translations (-Tr), rotations (-R) and all timebbased augmentations (-T). Experiments use the same training data, and we only change how we sample the positive pair (except for None and Depth experiments).
at each step, we keep the augmentation with the largest increase in accuracy and stop when there no more improvement. We find three main augmentations: rotations, crop and resize and color jittering. Adding other augmentations does not seem to have significant large effect on the accuracy.
To better understand the impact time-based augmentations during depth changes on the representa- tion, we also combine rotations with depth changes (R+D). Adding depth changes achieve similar performance to the addition of crop and resize. We hypothesize it mimics the "resize" part of the crop. An additional analysis (Figure (B) shows that the results are robust to the maximal speed of depth changes.
Virtual Home Environment. We test different sets of augmentations in the VHE, following the same procedure as Appendix B..2 In Figure TOA, we observe that rotations, ego-motion and color-based transformations are crucial for category recognition. Yet, can not conclude other augmentations are useless, since it may be the redundancy with other augmentations that is harmful, as we found with depth changes in Figure AA.
ToyBox environment. Figure 10 shows the impact of removing time-based augmentations in the BoyBox environment. We observe that removing augmentations based on translations (-Ho,Th) Tr),, and rotations (-Ho.R) both hurt the quality of the representation. Removing only some translations and translations (-Ho) do not affect the performance, presumably because other clips (with rotations and translations alone) already provide information about the shape of the object.
B.3 ROTATIONS ALLOW FOR BETTER SHAPE GENERALIZATION THAN THE FLIP AUGMENTATION
Here, we investigate how object rotations relate to the standard horizontal flip augmentation, since the flip resembles a 180-degrees rotation for symmetric objects and uniform background. In Figure ITA combine the flip augmentation and object rotations with other conventional augmentations. We see that when the object rotation speed is large enough (rot (360), there is no additional benefit in using the horizontal flip augmentation. However, rotations always boost performance compared to the pure flipping augmentation. Thus, we conclude that using 3-D object rotations rather than flipping results in considerably better representation of object shape.
B.4 HIGH-PREQUEECY OBJECT SWITCHES HURT CATEGORY RECOGNITION.
Here, study the impact of the object manipulation duration No. In 3DShapeE (Figure I1 B), observe a sweet spot at N, = 10 when using a simple convolutional neural network (ConvNet). This effect seems be architecture-dependeen since observe consistent increase of accuracy
16

---

Published as a conference paper at ICLR 2023
A
B
3DShapeE category acc.
3DShapeE category acc.
0.6
0.60
0.5
0.55
Accuracy
Accuracy
0.4
0.50
rot(360) flip rot(360) flip rot(90) flip rot(90)
ConvNet ResNet18
0.45
0.3
0.40
0.0
0.1
0.2 Million steps
0.3
2
10 Manipulation duration
20
50
Figure 11: A) 3DShapeB test category accuracy according to different combinations of flip and rotation speeds. B) 3DShapeE test category accuracy according to different manipulation duration No with simple ConvNet and a ResNet18.
increase No with a ResNet18. Importantly, in both cases, we observe that high-frequency object switches (low N.) hurt the downstream object categorization.
In CORe50 (T) the combined randomwalk (TT+ rw) method sensitive high frequency of object switches within the same session, which considerably improves validation set category accuracy (A). We assume that this because additional contrastive information is considerably higher for the randomwalk method, whereas uniform view sampling already achieves some degree of background variation due to sampling contrasts from one whole session compared to the neighboring frames. For test accuracies, observe similar trend, albeit not pronounced (B). Note how test-accuracies for the random walk procedure are considerably higher than for the uniform sampling.
B.5 INFLUENCE OF THE SESSION SWITCH FREQUENCY IN THE CORE50 ENVIRONMENT
Detailed analysis of varying the session frequency in the CORe50 environment. To complete the results from the CORe50 dataset add overview tables and a more thorough analysis regarding the randomwalk procedure. The mean and standard errors of the analysis corresponding to Figure pD can be found in Table together with a supervised and SimCLR baseline trained with the same hyperparameters. The uniform view sampling (uni) performs best on the validation set, whereas the two combined methods (*-TT+) deliver the best generalization capabilities across splits.
E [No]
algorithm
2
5
10
20
50
supervised
.530
.080
SimCLR
.544 + .030
-TT rw
.299 1 .018
.327 1: .033
.386 1 .046
.429 1. .055
.427 1. .052
-TT uni
.426 .055
.435 + .60
440 .056
.450 + .059
.440 + .056
-TT+ rw
.501 .032
.589 + .030
.615 .029
.629 + .031
.638 .032
-TT+ uni
.618 .084
.088 + .036
.620 .036
.616 + .036
.616 .033
Table Category accuracy and standard error on CORe50, based on SimCLR, P. 0.95. Standard eтror based on five different training/testing splits. Training occurred for 100 epochs, batchsize 512. Best performance is highlighted bold, cf. Figure
17

---

Published as a conference paper at ICLR 2023
A
val calegory acc.
test calegory acc.
C
val. session acc.
test session acc.
1.000
0.70
1.00
1.00
0.975
Accuracy
Accuracy
0.65
Accuracy
Aocuracy
0.950
0.98
0.60
0.96
0.925
0.98
TT+ TW TT+ uni
0.900
0.55
20
40
0
20
20
40
E[N,]
E[N,]
E[N,]
E[N,]
Figure 12: Analysis of the SimCLR-TT+ approach for different values of Ps, CORe50 Environment. Error bars indicate standard error based on three different training/testing splits.
We also report means and standard errors for category and session accuracy (Table 5 of ran- domwalk procedure when evaluated with different values for and Po-
Category accuracy
E [N.]
E [N.]
2
5
10
20
50
2
.993 + 0001
.994 + .001
.994 + .001
.997 .00
.998 + .00
5
.888 + .009
.974 + .002
.986 + .000
.989 .001
.992 + .001
10
.572 1 0005
.999 + .005
.956 1 .004
.972 .002
.911 1 .003
20
+ .004
.629 =
814 .010
.908 .009
.939 .004
50
.401 + 0077
.454 + .006
.522 + .007
.617 + .003
.711 .006
Session accuracy
E [N,]
E. [N.]
2
5
10
20
50
2
572 + .010
.535 + 0011
540 + .015
.507 + .008
.588 = 0000
5
.65 + .012
.644 + .014
.658 .018
.671
.011
.704 + 0011
10
.646 1 .016
.695 + .019
.118
.718 .014
.707 1 .009
20
.670 +
.673 =
.716 .021
.740 2013
.735 .013
50
.704 .012
.889 = .025
.732 - .012
.728
.731 = .220
Table 5: Category accuracy and standard error on CORe50, SimCLR-TT algorithm with randomwalk sampling procedure. Standard error based on three different training/testing splits. Training occurred for 100 epochs, batchsize 512, cf. Figure 5C, D.
Influence of session changes on combined augmentations. Since our main analysis with the CORe50 dataset aims give a solid overview over the presented approaches, we have picked medium level of session changes as the default (p. 0.95, E [N.pp.] 20). The results show that the combined approaches (*-TT+) outperform the conventional SimCLR method, but we were also interested in how sensitive the novel methods are to the Ps parameter. Results of that analysis are depicted in Figure 12] line with results from our Virtual Home Environment experiments, we observe that for very low values of N, the internal representation contains less information about the session for the randomwalk approach (C,D). The uniform method, however, shows robustness to this effect.
18

---

Published as a conference paper at ICLR 2023
C ASSETS
C.I 3D OBJECTS IN THE VIRTUAL HOME ENVIRONMENT
We import two versions of objects into the ThreeDWorld Software (TDW):
3DShapeE. This is the grey version of objects. We export the 3D models to .obj format and remove the texture files. We discard objects that took too long to process in TDW.
Textured objects. This is the colored version of objects. In order to import them in TDW, we apply series of operators: 1) we reduce the complexity of some meshes to make the next step computationally feasible and reduce the TDW processing complexity; 2) we manually bake most of the 3D models to obtain a single texture file with the meshes; 3) we import the models into TDW; 4) we resize the models marginalize the effect of size category recognition and avoid harmful collisions. We visually check the quality of all models. When we do not manage to obtain good-looking textures for some objects, remove them from dataset (< 15% of the dataset). The free category does not contain enough good-looking objects, so we remove the whole category.
C.2 TEST DATASETS OF THE VIRTUAL HOME ENVIRONMENT.
Our House test is composed of 7,740 images. from 1 1,548 objects distributed into 105 categories. By showcasing the category "elephant" in Figure 13 we exhibit the diversity of objects within one category. The elephant category includes flying elephants, bipedal elephants, red elephants, differently positioned clephants or geometrically simplified elephants. We also show the sequence of input images the VHE according to different rotation speeds.
In Figure 14, we display the three test houses used in Figure SB,C. House scenes were taken from pre-provided bundles TDW and we added the agent and objects in the scene similarly to Figure]
The underlying ThreeDWorld (TDW) software used is licenced under BSD 2- 2-Clause.
C.3 BoxBoX ENVIRONMENT
The ToyBox environment is a collection video-clips. We sample two frames per second from each clip, resulting in 159,627 images. Converted clips contain 2 50 sampled frames, which corresponds to the No parameter in the CORe50 environment. Two thirds of objects are used for the training set and the rest for the test set.
D HYPERPARAMETERS
For all conducted experiments, apply weight decay of 10-6 and update weights with the AdamW optimizer (Loshchilov and Hutter, 2018 and a learning rate of 10-4,
Virtual Home Environment and 3D Shape Environment. The agent in the Virtual Home En- vironment perceives the world around it as 128 x 128 pixel RGB images. Unless stated otherwise, these images are encoded by a succession of convolutional layers with the following [channels, kernel size, stride, padding] structure: [64, 8, 4, 2]. [128, 4, 2, 1], [256, 4, 2, 1]. [256, 4, 2. 1]. The output grouped by an average pooling layer and linear layer ending with 128 units. Each convolution layer is followed by non-linear ReLU activation function and a dropout layer (p 0.5) to prevent over-fitting, We not use projection heads. We consider a temperature hyperparameter SimCLR. We use batch size of 256 and a buffer size of 100,000. Average training time was 72 hours per run for VHE, and 20 hours for 3DShapeE. All experiments ran on GPUs of type NVIDIA V100.
0.1 for
We did a hyperparameter search of the best crop minimal size in {0.08, 0.2, 0.5, 0.75) for the Combined and SimCLR aug experiments.
CORe50 Dataset. The additional diversity of the CORe50 training data required more capable encoder network, which chose to be R8 8 (He al. 2016). The encoder transforms the
19

---

Published as a conference paper at ICLR 2023
Figure 13: All objects of the category "elephant" used our Virtual Home Environment.
B
Figure 14: Top view houses, without ceiling, used only when testing category recognition in unfamiliar houses experiments. A) House 2; B) House C) House 4.
20

---

Published as a conference paper at ICLR 2023
input images into 128-dimensione latent representation, which is followed by 2-layer linear projection head [256, 128]. For the projection head, we use batch normalization and ReLU activation function in-between the two layers. We chose batch size of 512 and training was done for 100 epochs on a GPU of type NVIDIA V100 or NVIDIA RTX2070 SUPER. Average training time was 17.5h per For B BOL in Bardes et al. (2022). The CORe50 dataset is licensed under CC-BY-4.0.
0.996 and for VICReg the default error weighting described
ToyBox environment. We use the same hyperparameters as for the CORe-50 experiments. How- ever, we made an ablation study of the crop minimal size hyperparameter to adapt to the small size and off-center position of the objects. We found a value of 0.5 be the best in (0.08, 0.2, 0.5, None).
21


Reviews:
[{"review_id": "d6OvifJLi1q", "reviewer": "Reviewer_fkWg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the idea of using time-based data augmentations to aid general purpose machine vision systems. Authors provide a thorough and very detailed psychological motivation for why this may work, and later use simulated agents's view points to simulate the time-based components of learning. Authors find that stacking time-based representations on top of other data-augmentation schemes aids in recognition.", "summary_of_the_review": "This is a type of paper that everyone talks about that \"someone will have to do this\", but no one is aware of anyone doing it, so I am happy that a paper regarding training a NN that incorporated the temporal dynamics in a SSL framework is studied and properly evaluated. This paper deserves an accept just because: 1) the authors have tried (and this isn't an obvious thing to do both in terms of finding an interesting scientific question and also the methodology, since most of the CV community is unfortunately still thinking about ImageNet, and developing a convoluted new architecture to increase the validation set performance by 0.1%); 2) authors find a set of interesting results and the cohesive story of the experimental design + evaluations mostly make a lot of senses (however, there's a couple of questions I have later on this).\n\nThe main **weakness** I find in this paper is that I think at some point there was some conversation (in the paper) about having a control condition where the video frames were shuffled as a control to eliminate the confounding variable of \"more data\". Without this control, the paper almost falls short because it still raises the question: **\"Do time-based augmentations drive this increase in performance due to their time-based nature, or by pure virtue of adding more training data?\"** Maybe authors have added these plots in the Appendix and I've missed this?\n\nOther than that, **this is a fantastic paper(!). I'm willing to increase my score if authors address my concerns in this review.**\n\n* Missing References:\n\n   * On eye-movements and multiple views for robustness, it may be worth adding the recent work of **Harrington & Deza. ICLR 2022** that showed through a set of psychophysical experiments that learning of multi-view foveal and peripheral template potentially aids in robust representations for humans (through adversarial robustness experiments in machines as well).\n\n   * Another key paper that is missing is the theme of time-based contrastive/self-supervised models are: ``Are models trained on temporally-continuous data streams more adversarially robust?'' by **Kong & Norcia. SVRHM 2021.**\n\nOf course, it would be quite interesting to see how time-based learning affects the agents performance in adversarial robustness and/or common corruptions. Perhaps this is not doable to do for the rebuttal (that is ok), but I'd be curious what the authors would think would happen. I expect stronger robustness, but will this actually be the case if some studies have shown that a robustness-accuracy tradeoff exists? (Tsipiras, ICLR 2019)", "strength_and_weaknesses": "See Summary of Review in section below. TLDR: The main paper's strengths are the original scientific question, and methodological execution (dataset, networks and learning regimes used). The main weakness is a tentative missing control to justify the main claim (see summary review section for details).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bUVUOZPnZNR", "reviewer": "Reviewer_UnyN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes and experiments with the idea of time-based augmentations: Human infants manipulate and move objects in front of themselves, move their eyes, making multiple fixations on an object, and carry objects into different rooms, providing independence of object appearance from background. All of these manipulations can be thought of as time-based augmentations of the data. Using these ideas with self-supervised learning provides a biologically-inspired way of learning representations of objects. The paper tests these ideas on three different self-supervised algorithms on four different datasets, and finds that adding time-based augmentations to standard ones significantly increases performance. The improvements are not in the 1-2% range that some papers show; rather, many results show a 10-20% increase, suggesting a real impact on self-supervised learning. Interestingly, time-based augmentations are not by themselves particularly effective (except in simulation); it is only in combination with standard augmentations that there is a substantial effect.\n\nI have read the other reviews (which I think are substantially unfair, and/or are from reviewers who, for some reason I fail to understand, want to kill the paper), the authors' rebuttals, and skimmed the updated paper. \n\nMy evaluation still stands: This is an excellent and exciting paper, well worthy of publication. It could be a good talk at the conference.", "summary_of_the_review": "This is an exciting paper, showing that \"natural\" temporal interactions with objects significantly boosts the performance of self-supervised algorithms. \n", "strength_and_weaknesses": "Strengths:\n\n+ The idea is well-motivated by data from infants\n+ The improvements are substantial, and impact three different categories of self-supervised systems.\n+ The analysis of the different factors affecting the outcome is fairly good.\n+ The method is tested in simulation, where the kinds of interaction can be controlled, and on real-world video of object manipulations.\n\nWeaknesses, with concrete, actionable feedback\n\n- The idea is not completely novel; several authors have proposed similar ideas. The difference here is the thoroughness of testing, the systematic analysis, and the very large datasets used, generated via simulation.\n\n- The paper is somewhat dense, due to all the experiments conducted, and is a little difficult to follow at times. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_GtKhINMbj", "reviewer": "Reviewer_Qi6u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to use temporal changes in object views that occur naturally in developing infants' visual environment as data augmentations for time-based self-supervised learning. Using a rendering environment they create positive and negative training pairs using 3d object manipulations such as rotations, saccadic eye movements, moving objects (to change the background). They compare to standard self-supervised learning using data augmentation and evaluate the learned representations on both the rendered and some real-world video datasets of humans manipulation objects.\n", "summary_of_the_review": "A well-motivated paper with creative and original aspects but relatively poor execution and somewhat unclear conclusions.\n\n### [Update after rebuttal]\n\nBased on the discussions and the other reviewers' impressions I increased my score and gave the paper the benefit of the doubt.", "strength_and_weaknesses": "### Strengths\n\n + Well motivated overall question\n + Important question how active vision could support visual representation learning beyond the simple data augmentation strategies currently used\n + Using rendering could be an interesting way forward to generate realistic view changes synthetically\n + Implemented an interesting and diverse set of view changes caused by active vision not captured by data augmentation\n\n\n### Weaknesses\n\n 1. Method does not actually result consistently in improved visual representations\n 1. Small size of datasets seems to be a major limiting factor preventing strong conclusions\n 1. Motivation for the individual experiments and how they each relate to the main research question is not very clear\n 1. Analyses are lacking depth\n\n\n\n### 1. No consistent improvement\n\nOne major concern is that the results do not really show clear evidence that the proposed approach indeed outperforms standard data augmentations. In more than half of the dataset/learning method combinations it actually performs worse.\n\n\n### 2. Small datasets\n\nAnother major concern is what we can actually learn about self-supervised learning in practically relevant scenarios from such small datasets. The fact that both methods (standard and time-based) combined work significantly better than each one individually suggests that in both cases the number of unique image sources/scenes is a major limiting factor and the combination of both methods simply leads to substantially more \"effective\" data. How each of the methods would fare in a more typical self-supervised setting where the datasets are large is not really answered by these experiments.\n\n\n### 3/4. Motivation + lack of depth\n\nThe authors present a collection of results, each of which addresses a question that is in principle interesting. However, it was not very clear to me why exactly these questions were asked and what we learn from the analyses. To name just one concrete example, why are saccades only simulated on CORe50 but not VHE? \n\nEach question is addressed very superficially by one manipulation, often on a single dataset, without discussing robustness and generality of these results: how much do they depend on the particular way the authors operationalized the question, what about different datasets or change in hyperparameters?\n\nOverall, after reading the paper I'm left wondering: what now? What follows for self-supervised learning? The fact that invariance to 3d rotations and changes in background are useful is undoubted, but such changes simply cannot be easily generated from unlabeled images and deriving them from unlabeled video is not a unique contribution of this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KVwLWRCEFm", "reviewer": "Reviewer_S6Vj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how augmentations induced by object interaction sessions help with self-supervised learning performance. This paper uses two synthetic environments and two real world data environments, each with different manipulation, ego-motion, and fixation related augmentations for self-supervised learning. The authors showed that these types of augmentations are more helpful than traditional image-space augmentations, and best if combined with those augmentations as well.\n\n", "summary_of_the_review": "In summary, I find this paper slightly below the bar of acceptance. My major concern is the notion of 'time-based augmentations', where a lot of those augmentations are just 3-D based and are time ignorant. To match this term more closely, the augmentations, or the positive frame selection must be time sensitive. To support the claim that this is helpful, the baseline should be a time-ignorant positive frame sampling strategy with the same set of augmentations. \nAlso, the types of augmentations shown here are marginally novel in pure SSL sense, but this could be an underestimate since the augmentations are designed more closely to mimic human responses.\n\n## Update after the rebuttal \nMy main concern is mostly around the term 'time-based': some of the augmentations used in this paper is not quintessential in terms of 'time'. One example is rotation of the object, which can be seen as an instance-centric 3D augmentation, where the ordering of different poses along the temporal dimension does not matter. \nThe longer phrase \"time-based augmentations during natural interactions\" is more accurate, and I apprecitate the change proposed by the authors. I do hope the title of the paper can be more accurate as well, but it is a relative minor issue.\nGiven the author's response, I would like to increase my rating from 5 to 6.", "strength_and_weaknesses": "#### Strength\n\nThis paper excels at performing well designed and controlled augmentations to study their performance in various settings. Through this design and careful control, it is convincing to show that the type of augmentation introduced are helpful for SSL.\n\n\n#### Weaknesses\n\nMy main concern for this paper is the notion of \"time-based\" augmentations. I find this term a bit misleading: most of the augmentation introduced seems time-invariant, but more as 3D augmentations. By time-invariant I mean in the case that if all the frames captured each session are randomly shuffled then piped to the SSL frameworks, augmentations like manipulation and ego motion seems unaffected at all, where all that matters are the set of positive and negative pairs, but not their temporal relations. Next frame positive pair might be affected, which is truly time-based. But according to [1], this may not be a good strategy for mining semantic correspondences. \n\nI therefore would like to ask the author for more arguments or evidence to support the use of \"time\", in addition to the 3D-based augmentations provided in the paper. Or a more proper name should be used. \n\n[1] Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster