Learnability and Expressiveness in Self-Supervised Learning

1 INTRODUCTION . We consider the unsupervised learning scenario , which aims at learning distributed representations for complex natural data without human annotations . Despite great recent progress in selfsupervised learning ( Chen et al. , 2020a ; Caron et al. , 2021 ; Grill et al. , 2020 ; Caron et al. , 2020 ; 2018 ; He et al. , 2020 ; Chen and He , 2021 ) , the problem of properly evaluating the quality of the learned representations has not been fully explored . We approach this question by drawing an analogy with the evolution of human language . Recent literature ( Swarup and Gasser , 2008 ; Smith et al. , 2013 ) suggests that linguistic structure arises as a trade-off between expressiveness and learnability . On one hand , expressiveness is the ability to discriminate objects and concepts of the world . On the other hand , Kirby et al . ( 2014 ) states that language evolution undergoes an iterated learning process , where each individual acquires language by observing a limited amount of language outputs of others ( Chomsky , 1980 ) . This process induces a form of compression , which makes the language simpler and more learnable for the next generation . Linguistic structure , e.g. , compositionality , emerges between these opposing forces . A similar guiding principle can be applied when learning representations of natural data . Many existing methods in self-supervised learning can be considered as attempts to increase expressiveness of the representations , i.e. , by maximizing or controlling the mutual information between inputs and representations , either through reconstruction ( Vincent et al. , 2008 ; Higgins et al. , 2017 ) or contrastive learning ( Oord et al. , 2018 ; Bachman et al. , 2019b ) . We argue that in addition to maximizing expressiveness , a good representation should also be learnable , e.g . support generalization from few examples ( Laina et al. , 2020 ) . In this paper , we propose an evaluation framework for self-supervised learning relying on expressiveness and learnability . Compared with previous work ( e.g. , Wang and Isola , 2020 ) , our framework is independent of ( i ) the availability of downstream task labels or predefined data augmentation techniques , and ( ii ) architecture choices and dimensionality of the latent representations , which makes it applicable to all previous SSL methods . We propose to measure expressiveness in terms estimators of the intrinsic dimension ( ID ) of the data representations ( Ansuini et al. , 2019 ) . Taking inspiration from Laina et al . ( 2020 ) , we quantify learnability in terms of the learning speed ( in terms of number of examples ) of a KNN learner trained to predict the K-Means cluster labels for held out representations . We dub this method Cluster Learnability ( CL ) . We show that high ID and CL correlate with high downstream performance for 30 different pretrained checkpoints across different architectures . To further demonstrate the usefulness of our framework , we present a case-study in which we improve upon DeepCluster ( Caron et al. , 2018 ) , which is a self-supervised learning method that iterates between clustering the representations and fitting the cluster labels . Inspired by our CL measure , we refine the cluster labels by executing KNN at the end of each K-means clustering round . The resulting method , KNN-DeepCluster , is able to outperform DeepCluster in both ImageNet and STL10 . When visualizing the training trajectories on the CL-ID plane , we find that our method indeed improves learnability at convergence . The correlation between performance and learnability holds also for checkpoints obtained during training , which suggests the possibility of using our framework to guide the development of new SSL algorithm , without labels at hand . Our contributions are summarized below : • We analyze self-supervised learning through the lens of expressiveness and learnability , and we propose tractable way to estimate them with Intrinsic Dimension ( ID ) and Cluster Learnability ( CL ) . • We carry out a large-scale evaluation on ImageNet and STL10 with 30 state-of-the-art selfsupervised and supervised checkpoints . We show that CL-ID is more correlated with the Top1 accuracy than previous approaches , with a Pearson ’ s r of 0.93 . • We propose a modification to DeepCluster to increase its cluster learnability . The resulting algorithm , KNN-DeepCluster is able to outperform DeepCluster . 2 RELATED WORK . Representation Evaluation Several recent works address the question of representation evaluation in self-supervised learning . Whitney et al . ( 2021 ) propose to use the learning dynamic of the downstream classifiers to measure the representation complexity , however the method depends on extra human labels . Wang and Isola ( 2020 ) decompose the contrastive loss into an alignment loss and a uniformity loss , which can be used to analyze the representation . However , their method requires access to the projection head output , limiting their applicability to general architecture . Pretext tasks like jigsaw , rotation prediction are shown to be well correlated with the supervised evaluation ( Reed et al. , 2021 ; Deng and Zheng , 2021 ) or architecture search ( Liu et al. , 2020 ) . However , as is shown in our experiments , they only provide limited predictive power when applying to a large pretrained family of checkpoints including both supervised and self-supervised models . In addition , these methods depends on predefined data augmentation , which might be task specific . Manifold Intrinsic Dimension Intrinsic dimension can be thought of as the smallest number of variable needed to approximate the representation manifold . Note that it is different from the intrinsic dimension of loss landscape ( Li et al. , 2018 ) , which measures minimum dimension of parameter space needed to optimize . We build upon extensive work on estimating intrinsic dimension . Applying local neighborhood information to estimate the intrinsic dimension is not a new idea , and it is shown to be more superior to the global eigenvalue approach ( Pettis et al. , 1979b ) . Ansuini et al . ( 2019 ) apply the TwoNN estimator ( Facco et al. , 2017 ) to the non-linear representation manifold of modern deep neural nets . They find that the intrinsic dimension is inversely correlated to the classification accuracy , and that the deep layers have smaller intrinsic dimensions . Their work is further extended to confirm that natural images lies in a low-dimension manifold ( Pope et al. , 2021 ) , and lower ID datasets leads to better generalization . Our work presents a more nuanced view about ID . It suggests that in order to learning representation from scratch , a least amount of ID is needed to convey the information from the dataset , e.g. , a constant-value representation has an ID of 0 , but no capacity to generalize at all . More discussion can be found in Section 5 . Learnability , Ease-of-Transmission and Compression Learnability has been argued to be a hallmark of the human language in order to be effortlessly transmitted through generations ( Kirby et al. , 2014 ; Rafferty et al. , 2011 ; Beckner et al. , 2017 ; Zhou and Yurovsky , 2021 ; Kampen , 2004 ) , and it is also true for visual concepts like color ( Xu et al. , 2010 ) , categories ( Griffiths et al. , 2006 ) , shapes ( Portelance et al. , 2021 ) etc . In deep learning , it has been explored in the context of emergent communication ( Ren et al. , 2020 ; Guo et al. , 2019 ; Li and Bowling , 2019 ) , language drift ( Lu et al. , 2020 ) , and neural module networks ( Vani et al. , 2021 ) , but it is less explored for vision representation learning , except for a human study on just two SSL methods Laina et al . ( 2020 ) . Learnability has a tight connection to compression ( Chaitin , 2007 ) and prequential codelength ( Dawid , 1984 ) , which quantifies the compression levels with the online learning error . Existing works ( Blier and Ollivier , 2018 ) has use it to support the generalization ability of the learner ( e.g. , deep neural nets ) on the dataset ( e.g. , labeled images ) . However , we use it to quantify the learnability of the representation , in the sense that if the emerged Kmeans clustering is more learnable , then the same KNN learner could achieve a lower compression bound via prequential coding . 3 PROPOSED FRAMEWORK . In this section , we introduce our setup and define our measures of expressiveness and learnability . 3.1 NOTATION . We consider the following setting : we assume we have a dataset of images { xi } Ni=1 represented in a pixel space X ⊂ Rd , where xi ∼ P are sampled i.i.d from some natural image distribution P over X . We also consider representation mapsF : X → Rm , typically pretrained neural networks , which represent any image as an m-dimensional vector . Any such map forms a representation dataset Z = { zi } Ni=1 where zi = F ( xi ) . Given zi ∈ Z and an integer k ≥ 1 , we denote by NN ( zi , k ) the k-th nearest neighbor of zi . While we generally mean ‘ nearest ’ with respect to Euclidean distance , in practice we will also use the cosine distance function,1 D ( z1 , z2 ) = 2− 2 cos ( z1 , z2 ) ( 1 ) Let rik = D ( zi , NN ( zi , k ) ) be the distance of the data point zi to its k-th nearest neighbor . 3.2 INTRINSIC DIMENSION ( ID ) . To measure expressiveness , we propose to use the notion of intrinsic dimensionality ( ID ) of the data in the representation space ( Pettis et al. , 1979a ) . Intuitively , as the representation becomes more expressive , we expect it to be able to represent images coming from more fine-grained categories , which could have multiple disentangled features such as color , shape , positions in space , etc . If these disentangled features correspond to different directions in some non-linear manifold , ID should yield a good surrogate for the expressiveness of the representation . Inferring the intrinsic dimension of a highly nonlinear manifold is a challenging problem ( e.g. , Levina and Bickel , 2005 ) . In this work , we leverage the nearest neighbor-based method of Facco et al . ( 2017 ) to estimate ID , which uses only the first two nearest neighbors of each point . This estimator ( TwoNN ) is shown to be reliable with respect to representation dimensions and scalable to real-world datasets with deep neural networks ( Ansuini et al. , 2019 ) . Assuming the data density is approximately constant around data points up to the distance of their neighbors , this method exploits the simple relationship between the distribution of the ratio of first and second nearest neighbor distances and the intrinsic dimension . Formally , let µi = ri2/ri1 be the ratio of distances for the data point zi . We suppose the points are sampled on a manifold with intrinsic dimension d. It can be shown that , under the above local uniformity assumption , µi follows a Pareto distribution with parameter d+ 1 on [ 1 , ∞ ) . That is P ( µ1 , · · · , µN |d ) = dN N∏ i=1 µ − ( d+1 ) i ( 2 ) While d can be computed by maximizing the likelihood , we follow a much simpler method proposed by Ansuini et al . ( 2019 ) and estimate d with a linear regression on the empirical cumulate of the distribution of µ , which is shown to have good practical performance ( Facco et al. , 2017 ) . This estimator is exact for uniformly distributed data . It is also shown to remain asymptotically correct for non-uniform sampling of zi when the image dataset is large enough ( Ansuini et al. , 2019 ) . 1Both distances produce similar results in our experiments .

Summary: This paper proposes Intrinsic Dimension (ID) and Cluster Learnability (CL) as an alternative solution to efficiently evaluate learned representation quality of pre-trained networks, without using down-stream tasks/labels. These two frameworks are inspired by finding “expressiveness” – finding the smallest number of variables to approximate the representation; and “learnability” – number of data samples needed for KNN clustering.  Extensive experiments have been evaluated on models trained with both supervised and unsupervised methods, and results showed that ID + CL would produce a high correlation with prediction accuracy from these methods, comparing to baselines. Finally, the authors argue that CL can also be incorporated into DeepCluster (an unsupervised method), as an auxiliary loss to further improve the prediction performance.