Contrastive Learning Through Time

1 INTRODUCTION . A hallmark of biological organisms is their ability to learn to understand the world around them in a largely autonomous fashion . Consider learning about visual objects . A human infant is not exposed to object views sampled i.i.d . from some fixed distribution and conveniently labeled image by image , but forms representations of objects and categories during extended interactions with individual objects ( Bambach et al. , 2018 ) and requires hardly any ( verbal ) labels for this ( LaTourrette & Waxman , 2019 ) . Mimicking such learning abilities in artificial systems would represent a giant leap forward for artificial intelligence . Self-supervised learning has emerged as a promising alternative to fully supervised approaches . In the domain of visual object recognition , recent contrastive learning approaches have obtained strong results on standard object recognition benchmarks ( Chen et al. , 2020a ; b ; Mitrovic et al. , 2021 ; Grill et al. , 2020 ) . These approaches rely on a range of so-called augmentation operations . The basic idea is that an image is transformed through a number of operations ( e.g. , scaling , flipping , cropping , rotating , blurring , color distortions , pixel noise , ... ) that change its appearance but not its meaning ( e.g. , “ cat ” ) . The key mechanism of contrastive learning is to form a representation where such augmented versions of an image are mapped on to close-by latent representations , while at the same time avoiding a “ collapse ” of the representation , i.e. , making sure the network does not simply map all inputs to the same point in the latent space . Not surprisingly , the quality of the learned representations for downstream tasks strongly depends on the chosen augmentation operations ( Grill et al. , 2020 ) . Theories of biological learning have also addressed the requirement to learn object representations without ( or with only few ) labels . The classic theory of how biological organisms learn invariant representations uses the notion of time to substitute for explicit labeling ( Földiák , 1991 ; Rolls & Milward , 2000 ; Wiskott & Sejnowski , 2002 ) . Biological organisms ( including the human infant mentioned above ) experience objects across time , typically seeing a sequence of different views of the same object before directing their attention elsewhere ( Fig . 1A ) . Thus , by learning a representation such that subsequent views are mapped onto close-by latent codes ( Fig . 1B ) , a representation should emerge that maps different views of the same object onto similar latent codes , thereby establishing ( partial ) invariance . While this idea has a long history in biological theorizing , it has only recently been explored in a contrastive learning context . The basic idea is to replace the augmentation operations in contrastive learning with natural appearance variation occurring during object interactions . To systematically study this approach , we propose a new Contrastive Learning Through Time ( CLTT ) framework that permits perfect control over the generated viewing sequences . For our experiments , we utilize the ThreeDWorld ( TDW ) virtual environment ( Gan et al. , 2021 ) , which allows near-photorealistic rendering . We also simulate classic biological experiments by Miyashita ( 1988 ) , demonstrating that objects form similar latent representations in the brain when they are systematically seen one after the other , even if they are visually dissimilar ( Miyashita , 1988 ) . We summarize our contributions as follows : • We develop the CLTT framework using state-of-the-art contrastive learning methods . • We introduce novel data sets to study CLTT under controlled conditions . • We systematically analyze the conditions for CLTT to be successful and demonstrate that it approaches fully supervised learning . • We show that CLTT maps objects that are systematically seen in temporal succession on similar latent representations , reminiscent of classic neurobiological findings . 2 RELATED WORK . An early demonstration that the temporal structure of visual inputs shapes object representations in primate visual cortex was given by Miyashita ( 1988 ) . He showed 97 images of fractal-like objects to monkeys in always the same order . As the monkeys learned to represent these images in their visual cortices , Miyashita ’ s observations suggested that the representations of objects which were neighbors in the sequence became aligned — even if these objects were visually dissimilar . This effect extended over a few objects , i.e. , the representations of objects six steps apart in the sequence were still more similar than the average similarity . In part motivated by such findings , there is a long history of neural network and machine learning models exploiting temporal structure for unsupervised representation learning . Földiák ( 1991 ) introduced so-called trace learning rules to explain how neurons in the mammalian visual system learn invariance properties , setting a starting point for later models considering multi-layered network architectures ( Rolls & Milward , 2000 ) . Another line of research introduced by Wiskott & Sejnowski ( 2002 ) explicitly considers the objective of extracting components from an input stream that are slowly changing . A recent variant attempts to do so in a biologically plausible fashion ( Lipshutz et al. , 2020 ) . The use of temporal learning objectives in contrastive learning has received increasing attention recently . Among the first , Mobahi et al . ( 2009 ) have proposed a method for learning object representations that combines supervised learning and unsupervised learning based on temporal coherence using a siamese neural network architecture . Subsequently , Wang & Gupta ( 2015 ) used tracking of patches in videos for unsupervised pre-training , by learning an embedding that keeps patches from the same track close in the embedding space . An approach more closely related to ours has recently been proposed by Orhan et al . ( 2020 ) . They consider learning on a longitudinal headcam video set from three developing children ( Sullivan et al. , 2020 ) . They focus on a temporal classification approach , where they divide the videos into a finite number of contiguous segments of the same length that they call temporal classes . The learning objective is to predict from which of the classes a particular video frame originates . They also consider a temporal contrastive learning objective with the MoCo contrastive learning implementation of Chen et al . ( 2020b ) . This objective also aims to make the latent representations of adjacent video frames similar . However , the use of uncontrolled headcam video does not permit determining the required the temporal statistics of the visual input for the approach to work . Another related approach is that of Knights et al . ( 2021 ) , who learn embeddings of video clips . Their learning objective makes latent codes of adjacent frames within a video clip similar , while making them distinct from latent codes of frames from other video clips . Note that this setup requires the video clips to be segmented , i.e. , the system has access to the information where each video starts and ends rather than being exposed to an unlabeled continuous video stream as in Orhan et al . ( 2020 ) and CLTT . The same holds true for the recent approach of Feichtenhofer et al . ( 2021 ) and also Pan et al . ( 2021 ) . Yet , they don ’ t make the connection between continuous time frames and biologically-inspired vision . Finally , the interesting work of Stojanov et al . ( 2019 ) has modeled continual infant-like learning of object representations from unsegmented input streams , but they did not consider a contrastive learning approach . 3 METHODS . 3.1 SAMPLING SEQUENCES OF OBJECT VIEWS IN CLTT . CLTT aims to mimic the stream of object views that an infant may experience during natural interactions with objects , while giving precise control over the statistical properties of this view sequence . Specifically , an object is always viewed for Nfix fixations before another object comes into view . Importantly , the learner does not have access to the information when a new object comes into view , making the approach fully unsupervised . Generating these view sequences involves two sampling procedures . The first describes how the next view of the same object is sampled during the Nfix fixations on the same object . We refer to this as view sampling . The second describes how the identity of the next object is determined at the end of the Nfix fixations on the same object , which we refer to as object sampling . For the view sampling , we distinguish two sampling methods . Our default method is the random walk view sampling . Here , the next view of an object is a “ neighbor ” of the previous view . For example , in the TDW data set ( see below ) this corresponds to changing azimuth or elevation by 10◦ or viewing distance by 10 cm . This procedure mimics the infant gradually turning an object or moving around an object while fixating it . The second method is the uniform view sampling . Here , the next view is picked uniformly at random . While this does not mimic infants ’ viewing sequences , it provides more diversity among successive views , which is expected to aid learning . For the object sampling we also distinguish two methods . Our default method is the random order method . Here , the order of the objects is a new random permutation during each training cycle . This corresponds to the case that the infant encounters objects randomly . The second method uses a fixed order of objects , i.e. , object A is always followed by object B , etc . in all training cycles . This situation matches the neurobiological experiments by Miyashita ( 1988 ) and is expected to lead to an alignment of the latent codes of objects that are consistently seen in succession . Training occurs in cycles . In every cycle each of the Nobj objects will be chosen exactly once — the order depending on the chosen object sampling method . Thus , a cycle consists of a total of Nobj × Nfix object views . The Nfix views for each object are determined by the chosen view sampling procedure . Several cycles of Nobj×Nfix views are stored in a buffer . For our experiments we use a batch size of 256 . During learning , batches of pairs of subsequent views are sampled uniformly from the buffer . This leads to a probability 1/Nfix of sampling inter-class transitions and a probability of ( Nfix−1 ) /Nfix of sampling intra-class transitions ( compare Fig . 1 ) . We consider an epoch to be a full run through the buffer , which is chosen to be of equal size as the underlying data set . That means that on average , every image of the data set is presented once during each epoch . 3.1.1 CONTRASTIVE LEARNING ALGORITHMS FOR CLTT . The sampled sequences can be fed into a wide range of contrastive learning methods . Here we consider SimCLR ( Chen et al. , 2020a ) and RELIC ( Mitrovic et al. , 2021 ) in which for every positive pair , all the other pairs in the batch are considered negative pairs . We also experiment with BYOL ( Grill et al. , 2020 ) , that uses only positive pairs . We refer to the CLTT versions of these algorithms as SimCLR-TT , RELIC-TT , and BYOL-TT , respectively . In all cases , we use a ResNet-18 architecture ( He et al. , 2016 ) to transform the input images into 128-dimensional latent representations , which is followed by a single layer as the projection head for RELIC-TT and SimCLR-TT . For BYOL-TT we use a two layer projection head with batch normalization after the hidden layer . SimCLR-TT . SimCLR ( Chen et al. , 2020a ) is an effective contrastive learning approach for visual representation learning . By defining a wide range of augmentation operations and treating differently augmented images as positive samples , SimCLR has achieved state-of-the-art performance . Its groundbreaking success has triggered a surge of interest in augmentation-based self-supervised learning . For SimCLR-TT we replace the traditional augmentations with successive views as pairs defined in the previous subsection . For each positive pair , all remaining pairs in the batch are considered negative samples . The loss of SimCLR-TT is then defined as follows : L ( zi ) = − log exp ( sim ( zi , z ′ i ) /τ ) ∑NB k=1 , k ̸=i [ exp ( sim ( zi , zk ) ) + exp ( sim ( zi , z ′ k ) ) ] /τ , ( 1 ) where zi and z′i are the latent codes of a sampled view pair , and NB is the number of pairs in a batch . Specifically , we use the cosine similarity as the similarity function sim ( u , v ) = u ⊤v ∥u∥∥v∥ . For simplicity , we set the temperature parameter τ in the original SimCLR loss to be 1 . In all our experiments , we use the AdamW optimizer ( Loshchilov & Hutter , 2019 ) with a starting learning rate of 10−3 , which decays by a factor of 0.3 after every 10 epochs . The batch size for training is 256 and the buffer size is determined by the size of the data set . RELIC-TT . RELIC ( Mitrovic et al. , 2021 ) is an approach that uses an additional penalty loss compared to SimCLR . It obtains state-of-the-art results by keeping the similarity distribution of one sample invariant against differently augmented views of other samples . Incorporating this notion into the CLTT approach , we derive RELIC-TT by adding another loss term that . The loss can be described as follows : Lp ( zi ) = KL ( p ( Y | zi ) , p ( Y | z′i ) ) , ( 2 ) where KL is the Kullback-Leibler divergence and p ( Y = j | zi ) = exp ( sim ( zi , z ′ j ) ) ∑NB k=1 exp ( sim ( zi , z ′ k ) ) . ( 3 ) In the definition above , zi and z′i are the latent codes of the first view and the second view of the sampled view pair . The similarities between zi and all the second views in every pair are calculated and passed to a Softmax function as shown in equation 3 . The same computation is done for the second view z′i for all the other first views , resulting in totally two probability distributions denoted as p ( Y | zi ) and p ( Y | z′i ) that resembles a classification task of predicting the label Y out of totally NB classes . And then those two distributions are pulled together by minimizing the KL divergence between them . The training configuration is the same as used in SimCLR-TT . BYOL-TT . BYOL-TT builds on the Bootstrap Your Own Latent ( BYOL ) architecture by ( Grill et al. , 2020 ) . BYOL has been shown to outperform other contrastive learning architectures like SimCLR ( Chen et al. , 2020a ) or MoCov2 ( Chen et al. , 2020b ) on the ImageNet data set ( Deng et al. , 2009 ) . A key advantage of the BYOL architecture is that it works without negative pairs , which sets it apart from other contrastive learning algorithms . Instead , it uses a second so-called target network . The target network receives an augmented version of the input data like the online network . It will produce a target projection and the online network tries to predict this target projection . The loss function minimizes the similarity between the target projection and the prediction of the online network . In our BYOL-TT implementation , we use the AdamW optimizer with a learning rate of 2× 10−4 which decays by a factor of 0.3 after every 30 epochs and a batch size of 256 . The loss is given by : Lθ , ξ = 2− 2 · ⟨qθ ( zθ ) , z′ξ⟩ ||qθ ( zθ ) ||2 · ||z′ξ||2 , ( 4 ) where qθ ( zθ ) is the prediction of the online network of z′ξ . θ corresponds to the weights of the online network and ξ represent the weights of the target network . The optimization will be performed with respect to θ , the weights ξ of the target network will be updated using an exponential moving average of the online network . Here we choose τ = 0.99 as target decay rate . In the original BYOL architecture , an input image x produces two augmented views v and v′ which are then shown to the target and online network . In our case no augmentations are applied to the input image , positive pairs are formed by images next to each other in the presented data timeline . Two consecutive images in this timeline will then be used as views v and v′ .

Summary: The paper proposes a new method to select positive pairs for contrastive learning: using adjacent frames in a sequence, referred to as Contrastive Learning Through Time (CLTT). This is relevant since CL approaches have been shown to depend a lot on what kind of augmentation is used. The method is integrated with a number of well-known CL approaches, and evaluated on three different datasets, two of which are novel --the method is shown to approach fully supervised results. The paper also provides a valuable discussion on how the notion of time can help representation learning. The method is also tested for latent representation similarity when one object or pattern is systematically shown after another, something which has been studied in the field of biological learning, and a similar trend is found here as well.