Disentangling Properties of Contrastive Methods

1 INTRODUCTION . Learning a disentangled representation is a long-desired goal in the deep learning community ( Bengio et al. , 2013 ; Peters et al. , 2017 ; Goodfellow et al. , 2016 ; Bengio et al. , 2007 ; Schmidhuber , 1992 ; Lake et al. , 2017 ; Tschannen et al. , 2018 ) . A disentangled representation matches how humans understand the world and provides many other benefits besides model interpretability ( Bengio et al. , 2013 ; Chen et al. , 2016 ; Kulkarni et al. , 2015 ) . And it usually needs much fewer labels to learn challenging downstream tasks ( van Steenkiste et al. , 2019 ) . It also generalizes much better even in face of examples generated by an unseen combination of the attribute values ( Achille et al. , 2018 ) . Given its importance and potential large impacts on downstream applications , disentangled representation learning has recently attracted great attention . Previous research has proposed a lot of methods , either built on variational auto-encoders ( Kingma & Welling , 2013 ) , such as β-VAE ( Higgins et al. , 2016 ) and FactorVAE ( Kim & Mnih , 2018 ) , or generative adversarial network ( Goodfellow et al. , 2014 ) , such as InfoGAN ( Chen et al. , 2016 ) and InfoGAN-CR ( Lin et al. , 2020 ) . Those methods have achieved preliminary successes on synthetic datasets such as dSprites ( Matthey et al. , 2017 ) and 3Dshapes ( Burgess & Kim , 2018 ) . Albeit the initial successes , those synthetic datasets are limited in many aspects , for example , the background is usually clean and composed of a single color , the number of objects is small and the objects are mainly 2D without texture and occlusions . It is still an open question to scale those models to real-world complex datasets . It is even non-trivial to scale basic generative models to complex datasets to learn disentangled features , such as ImageNet ( Deng et al. , 2009 ) . In this paper , instead of studying disentanglement feature learning with a generative model , we investigate whether the discriminatively-trained contrastive models have the disentanglement property . Contrastive learning is a class of self-supervised learning methods that pull two augmentations of the same image close . The recent contrastive methods have achieved state-of-the-art performance on the image pretraining tasks . Representative methods include MoCo ( He et al. , 2020 ) , BYOL ( Grill et al. , 2020 ) , SimCLR ( Chen et al. , 2020 ) and SwAV ( Caron et al. , 2020 ) , etc . Contrastive learning has been proven to learn good visual representations from large-scale datasets . We continue to investigate the disentanglement property of their learned representations in this work . To our surprise , we find that the widely used BYOL algorithm without any auxiliary loss exhibits strong feature disentanglement property . However , the disentanglement of contrastive learning is a weaker form of disentanglement . It follows the pattern that a representation dimension is disentangled to correspond to a single factor but a single ground truth factor might appear in multiple latent feature dimensions . We name this type of disentanglement as “ group disentanglement ” . Although group disentanglement is a weaker form of disentanglement , we hypothesize that directly learning a compact and disentangled representation might be hard , due to the lottery ticket hypothesis ( Frankle & Carbin , 2018 ) . Pursuing group disentanglement instead of full disentanglement might be necessary to achieve disentanglement on the complex real-world dataset . The reason is probably that real-world images usually contain more details and noise , and a “ factor ” might be always correlated to some other visual existence due to dataset bias . A good example is a ship that usually comes together with a large area of blue background in existing datasets . Further , we find that contrastively trained representation achieves the state-of-the-art FactorVAE disentanglement score when evaluated on established benchmarks , such as Car3D ( Reed et al. , 2015 ) , dSprites ( Matthey et al. , 2017 ) and SmallNORB ( LeCun et al. , 2004 ) . Besides the widely used synthetic benchmarks , we also evaluate the contrastive method on the CelebA ( Liu et al. , 2018 ) human faces real-world dataset . We find that it also achieves better or comparable performances than the other methods on five commonly used disentanglement metrics . In summary , our contributions in this paper is mainly empirical and include the follows : 1 . We show that a contrastive method , in particular BYOL , learns “ group-disentangled ” representations , without any extra auxiliary losses . 2 . The contrastive method achieves the state-of-the-art performance on several widely used disentanglement learning benchmarks . 3 . We propose to quantitatively evaluate disentanglements on a real-world dataset , which avoids the biases of synthetic images . Our contrastive method also achieves state-of-the-art performance on this benchmark . 2 RELATED WORKS . Disentangled Representation Learning Disentangled representation is desired as it represents a human interpretable pattern ( Bengio et al. , 2013 ; Chen et al. , 2016 ; Kulkarni et al. , 2015 ) , enabling the downstream tasks learned more easily ( van Steenkiste et al. , 2019 ) and generalizes better ( Achille et al. , 2018 ) . In this paper , we consider the fully unsupervised disentangled representation learning setting , i.e . we assume no annotations on which factors should be learned . We notice the recent study of disentanglement is promoted by two communities : Disentanglement in Deep Features and Independent Component Analysis . Their research previously lie on different assumptions , data patterns , and evaluation metrics . One community is motivated by the newly raised deep learning for encouraging disentangled representation over independent factors . they have shown much empirical progress on this problem and they directly term their goal as “ disentanglement ” . The related study is usually based on deep generative models . For instance , VAE-based methods have achieved successes on this task ( Higgins et al. , 2016 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Kumar et al. , 2017 ) . Besides , Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ; Chen et al. , 2016 ) are also put into the discussion of encouraging representations ’ disentanglement . More recently , people have shown that the GAN-based approach can achieve competitive performance as the above VAE variants ( Lin et al. , 2020 ; Jeon et al. , 2018 ; Lee et al. , 2020b ) . A recent work ( Locatello et al. , 2019 ) summarizes the popular methods and metrics in this community and proposes a tool for evaluation called disentanglement lib , including popular metrics such as DCI ( Eastwood & Williams , 2018 ) , SAP ( Kumar et al. , 2017 ) , MIG ( Chen et al. , 2018 ) and so on . Besides this series of studies , exploring underlying factors of variation in data pattern is a longstanding goal of the Independent Component Analysis ( ICA ) community ( Hyvärinen & Oja , 2000 ) . They share many similarities , for example , generative models , e.g. , VAEs , are recently popular in both ( Khemakhem et al. , 2020a ; Klindt et al. , 2020 ) . ICA usually has different assumptions with the “ purely unsupervised learning ” ( Hälvä et al. , 2021 ) . For example , the pattern of noise ( Hyvarinen & Morioka , 2016 ; Khemakhem et al. , 2020a ) or some additional auxiliary variables ( Hyvarinen et al. , 2019 ; Khemakhem et al. , 2020b ) can be observed . Traditionally , ICA uses identifiability to assess their desired representation pattern and the popular metric is Mean Correlation Coefficient ( MCC ) . SlowVAE ( Klindt et al. , 2020 ) recently makes a great effort to connect the two branches of study but it still requires additional information such as temporal transition pattern . As our study is for purely unsupervised learning and some assumptions of ICA methods can not be well matched , we mainly follow the settings and benchmark by disentanglement libk ( Locatello et al. , 2019 ) . Contrastive Learning Contrastive learning methods such as SimCLR ( Chen et al. , 2020 ) , MoCo ( He et al. , 2020 ) and BYOL ( Grill et al. , 2020 ) have achieved great successes to learn good visual representation from no label . They create “ views ” by applying augmentations over images . They treat two views of the same image as “ positive pairs ” , and views of all the other images as negatives . This setup is also known as examplar classification ( Dosovitskiy et al. , 2014 ) , or instance discrimination ( Wu et al. , 2018 ) . Representation learned by contrastive learning has shown great transfer capabilities to downstream tasks , such as object detection and semantic segmentation . More recently , there are a lot of works trying to understand contrastive learning either theoretically ( Wang & Isola , 2020 ; Arora et al. , 2019 ; Tsai et al. , 2020 ; Tosh et al. , 2021 ; Tian et al. , 2020b ; Lee et al. , 2020a ) or empirically ( Tian et al. , 2020a ; Zhao et al. , 2020 ; Purushwalkam & Gupta , 2020 ) . Zimmermann et al . ( 2021 ) suggests that the contrastive method can invert the data generation process . The conclusion is based on the analysis of Wang & Isola ( 2020 ) where negative samples are necessary and expected to be infinite . Zimmermann et al . ( 2021 ) make a good bridge between contrastive learning and independent analysis and study the model identifiability quantitatively in terms of MCC score . Compared with that , we focus more on more direct disentanglement analysis . Our contribution is more empirical but suggests the good disentanglement property of contrastive learning even without negative samples . 3 METHOD . In this paper , we explore whether contrastive methods learn a disentangled feature representation . If yes , under what condition it learns a disentangled representation . There are quite a few contrastive learning algorithms proposed ( Grill et al. , 2020 ; He et al. , 2020 ; Chen et al. , 2020 ; Caron et al. , 2020 ) . Although they differ on some specific aspects , they all aim to pull two augmentations of one image close . Without the uniformity property provided by negative samples ( Wang & Isola , 2020 ) , we find BYOL ( Grill et al. , 2020 ) still achieves unexpected good disentangled feature representation . 3.1 BYOL METHOD . BYOL is an unsupervised learning method that pulls two augmentation views of the same image close ( Grill et al. , 2020 ) to learn a high-level image representation . A significant difference of it against other contrastive learning is the absence of negative pairs during training . As shown in Figure 1 , for each image x , we obtain two views of it : x1 and x2 by data augmentations . One of them goes through the online network stream , and the other goes through the target network stream . The target network ’ s parameter is not trained by the gradient descent algorithm but set as the exponential moving average ( ema ) of the online network . Both the online and the target stream have a representation network ( encoder ) and a projection network . The online network has an extra prediction network after the projection network . The online stream ’ s output z1 and the target stream ’ s output z2 are pulled close to each other by requiring the two vectors to have similar directions in the latent space . More specifically , the loss function is L = − 〈z1 , z2〉 ‖z1‖2 ‖z2‖2 . In practice , the online representation network ( without the projection or the prediction network ) is usually the representation model for the downstream tasks . In this work , we follow the same convention , i.e . we study the disentanglement property of the output of the representation network .

Summary: This paper makes an interesting empirical observation - BYOL representations have better disentanglement properties, according to some metrics, than current specialized methods. Furthermore, the authors found that the selection of the normalization function affects the results. The authors also claim the dimensions are "group disentangled" although this is only shown on one dataset.