Disentangling Properties of Contrastive Methods

1 INTRODUCTION . Learning a disentangled representation is a long-desired goal in the deep learning community ( Bengio et al. , 2013 ; Peters et al. , 2017 ; Goodfellow et al. , 2016 ; Bengio et al. , 2007 ; Schmidhuber , 1992 ; Lake et al. , 2017 ; Tschannen et al. , 2018 ) . A disentangled representation matches how humans understand the world and provides many other benefits besides model interpretability ( Bengio et al. , 2013 ; Chen et al. , 2016 ; Kulkarni et al. , 2015 ) . And it usually needs much fewer labels to learn challenging downstream tasks ( van Steenkiste et al. , 2019 ) . It also generalizes much better even in face of examples generated by an unseen combination of the attribute values ( Achille et al. , 2018 ) . Given its importance and potential large impacts on downstream applications , disentangled representation learning has recently attracted great attention . Previous research has proposed a lot of methods , either built on variational auto-encoders ( Kingma & Welling , 2013 ) , such as β-VAE ( Higgins et al. , 2016 ) and FactorVAE ( Kim & Mnih , 2018 ) , or generative adversarial network ( Goodfellow et al. , 2014 ) , such as InfoGAN ( Chen et al. , 2016 ) and InfoGAN-CR ( Lin et al. , 2020 ) . Those methods have achieved preliminary successes on synthetic datasets such as dSprites ( Matthey et al. , 2017 ) and 3Dshapes ( Burgess & Kim , 2018 ) . Albeit the initial successes , those synthetic datasets are limited in many aspects , for example , the background is usually clean and composed of a single color , the number of objects is small and the objects are mainly 2D without texture and occlusions . It is still an open question to scale those models to real-world complex datasets . It is even non-trivial to scale basic generative models to complex datasets to learn disentangled features , such as ImageNet ( Deng et al. , 2009 ) . In this paper , instead of studying disentanglement feature learning with a generative model , we investigate whether the discriminatively-trained contrastive models have the disentanglement property . Contrastive learning is a class of self-supervised learning methods that pull two augmentations of the same image close . The recent contrastive methods have achieved state-of-the-art performance on the image pretraining tasks . Representative methods include MoCo ( He et al. , 2020 ) , BYOL ( Grill et al. , 2020 ) , SimCLR ( Chen et al. , 2020 ) and SwAV ( Caron et al. , 2020 ) , etc . Contrastive learning has been proven to learn good visual representations from large-scale datasets . We continue to investigate the disentanglement property of their learned representations in this work . To our surprise , we find that the widely used BYOL algorithm without any auxiliary loss exhibits strong feature disentanglement property . However , the disentanglement of contrastive learning is a weaker form of disentanglement . It follows the pattern that a representation dimension is disentangled to correspond to a single factor but a single ground truth factor might appear in multiple latent feature dimensions . We name this type of disentanglement as “ group disentanglement ” . Although group disentanglement is a weaker form of disentanglement , we hypothesize that directly learning a compact and disentangled representation might be hard , due to the lottery ticket hypothesis ( Frankle & Carbin , 2018 ) . Pursuing group disentanglement instead of full disentanglement might be necessary to achieve disentanglement on the complex real-world dataset . The reason is probably that real-world images usually contain more details and noise , and a “ factor ” might be always correlated to some other visual existence due to dataset bias . A good example is a ship that usually comes together with a large area of blue background in existing datasets . Further , we find that contrastively trained representation achieves the state-of-the-art FactorVAE disentanglement score when evaluated on established benchmarks , such as Car3D ( Reed et al. , 2015 ) , dSprites ( Matthey et al. , 2017 ) and SmallNORB ( LeCun et al. , 2004 ) . Besides the widely used synthetic benchmarks , we also evaluate the contrastive method on the CelebA ( Liu et al. , 2018 ) human faces real-world dataset . We find that it also achieves better or comparable performances than the other methods on five commonly used disentanglement metrics . In summary , our contributions in this paper is mainly empirical and include the follows : 1 . We show that a contrastive method , in particular BYOL , learns “ group-disentangled ” representations , without any extra auxiliary losses . 2 . The contrastive method achieves the state-of-the-art performance on several widely used disentanglement learning benchmarks . 3 . We propose to quantitatively evaluate disentanglements on a real-world dataset , which avoids the biases of synthetic images . Our contrastive method also achieves state-of-the-art performance on this benchmark . 2 RELATED WORKS . Disentangled Representation Learning Disentangled representation is desired as it represents a human interpretable pattern ( Bengio et al. , 2013 ; Chen et al. , 2016 ; Kulkarni et al. , 2015 ) , enabling the downstream tasks learned more easily ( van Steenkiste et al. , 2019 ) and generalizes better ( Achille et al. , 2018 ) . In this paper , we consider the fully unsupervised disentangled representation learning setting , i.e . we assume no annotations on which factors should be learned . We notice the recent study of disentanglement is promoted by two communities : Disentanglement in Deep Features and Independent Component Analysis . Their research previously lie on different assumptions , data patterns , and evaluation metrics . One community is motivated by the newly raised deep learning for encouraging disentangled representation over independent factors . they have shown much empirical progress on this problem and they directly term their goal as “ disentanglement ” . The related study is usually based on deep generative models . For instance , VAE-based methods have achieved successes on this task ( Higgins et al. , 2016 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Kumar et al. , 2017 ) . Besides , Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ; Chen et al. , 2016 ) are also put into the discussion of encouraging representations ’ disentanglement . More recently , people have shown that the GAN-based approach can achieve competitive performance as the above VAE variants ( Lin et al. , 2020 ; Jeon et al. , 2018 ; Lee et al. , 2020b ) . A recent work ( Locatello et al. , 2019 ) summarizes the popular methods and metrics in this community and proposes a tool for evaluation called disentanglement lib , including popular metrics such as DCI ( Eastwood & Williams , 2018 ) , SAP ( Kumar et al. , 2017 ) , MIG ( Chen et al. , 2018 ) and so on . Besides this series of studies , exploring underlying factors of variation in data pattern is a longstanding goal of the Independent Component Analysis ( ICA ) community ( Hyvärinen & Oja , 2000 ) . They share many similarities , for example , generative models , e.g. , VAEs , are recently popular in both ( Khemakhem et al. , 2020a ; Klindt et al. , 2020 ) . ICA usually has different assumptions with the “ purely unsupervised learning ” ( Hälvä et al. , 2021 ) . For example , the pattern of noise ( Hyvarinen & Morioka , 2016 ; Khemakhem et al. , 2020a ) or some additional auxiliary variables ( Hyvarinen et al. , 2019 ; Khemakhem et al. , 2020b ) can be observed . Traditionally , ICA uses identifiability to assess their desired representation pattern and the popular metric is Mean Correlation Coefficient ( MCC ) . SlowVAE ( Klindt et al. , 2020 ) recently makes a great effort to connect the two branches of study but it still requires additional information such as temporal transition pattern . As our study is for purely unsupervised learning and some assumptions of ICA methods can not be well matched , we mainly follow the settings and benchmark by disentanglement libk ( Locatello et al. , 2019 ) . Contrastive Learning Contrastive learning methods such as SimCLR ( Chen et al. , 2020 ) , MoCo ( He et al. , 2020 ) and BYOL ( Grill et al. , 2020 ) have achieved great successes to learn good visual representation from no label . They create “ views ” by applying augmentations over images . They treat two views of the same image as “ positive pairs ” , and views of all the other images as negatives . This setup is also known as examplar classification ( Dosovitskiy et al. , 2014 ) , or instance discrimination ( Wu et al. , 2018 ) . Representation learned by contrastive learning has shown great transfer capabilities to downstream tasks , such as object detection and semantic segmentation . More recently , there are a lot of works trying to understand contrastive learning either theoretically ( Wang & Isola , 2020 ; Arora et al. , 2019 ; Tsai et al. , 2020 ; Tosh et al. , 2021 ; Tian et al. , 2020b ; Lee et al. , 2020a ) or empirically ( Tian et al. , 2020a ; Zhao et al. , 2020 ; Purushwalkam & Gupta , 2020 ) . Zimmermann et al . ( 2021 ) suggests that the contrastive method can invert the data generation process . The conclusion is based on the analysis of Wang & Isola ( 2020 ) where negative samples are necessary and expected to be infinite . Zimmermann et al . ( 2021 ) make a good bridge between contrastive learning and independent analysis and study the model identifiability quantitatively in terms of MCC score . Compared with that , we focus more on more direct disentanglement analysis . Our contribution is more empirical but suggests the good disentanglement property of contrastive learning even without negative samples . 3 METHOD . In this paper , we explore whether contrastive methods learn a disentangled feature representation . If yes , under what condition it learns a disentangled representation . There are quite a few contrastive learning algorithms proposed ( Grill et al. , 2020 ; He et al. , 2020 ; Chen et al. , 2020 ; Caron et al. , 2020 ) . Although they differ on some specific aspects , they all aim to pull two augmentations of one image close . Without the uniformity property provided by negative samples ( Wang & Isola , 2020 ) , we find BYOL ( Grill et al. , 2020 ) still achieves unexpected good disentangled feature representation . 3.1 BYOL METHOD . BYOL is an unsupervised learning method that pulls two augmentation views of the same image close ( Grill et al. , 2020 ) to learn a high-level image representation . A significant difference of it against other contrastive learning is the absence of negative pairs during training . As shown in Figure 1 , for each image x , we obtain two views of it : x1 and x2 by data augmentations . One of them goes through the online network stream , and the other goes through the target network stream . The target network ’ s parameter is not trained by the gradient descent algorithm but set as the exponential moving average ( ema ) of the online network . Both the online and the target stream have a representation network ( encoder ) and a projection network . The online network has an extra prediction network after the projection network . The online stream ’ s output z1 and the target stream ’ s output z2 are pulled close to each other by requiring the two vectors to have similar directions in the latent space . More specifically , the loss function is L = − 〈z1 , z2〉 ‖z1‖2 ‖z2‖2 . In practice , the online representation network ( without the projection or the prediction network ) is usually the representation model for the downstream tasks . In this work , we follow the same convention , i.e . we study the disentanglement property of the output of the representation network .

Summary: This work explores the disentangling properties of contrastive methods. The authors discover that contrastive methods, particularly BYOL, learns disentangled representations with just a change of normalization method in the encoder. The work also proposes a new concept called "group disentanglement", which is a relaxed version of the original disentanglement. BYOL learns representations with group disentanglement and achieves SOTA on disentanglement benchmarks on not only synthetic image datasets but also a real world dataset.