Learnability and Expressiveness in Self-Supervised Learning

1 INTRODUCTION . We consider the unsupervised learning scenario , which aims at learning distributed representations for complex natural data without human annotations . Despite great recent progress in selfsupervised learning ( Chen et al. , 2020a ; Caron et al. , 2021 ; Grill et al. , 2020 ; Caron et al. , 2020 ; 2018 ; He et al. , 2020 ; Chen and He , 2021 ) , the problem of properly evaluating the quality of the learned representations has not been fully explored . We approach this question by drawing an analogy with the evolution of human language . Recent literature ( Swarup and Gasser , 2008 ; Smith et al. , 2013 ) suggests that linguistic structure arises as a trade-off between expressiveness and learnability . On one hand , expressiveness is the ability to discriminate objects and concepts of the world . On the other hand , Kirby et al . ( 2014 ) states that language evolution undergoes an iterated learning process , where each individual acquires language by observing a limited amount of language outputs of others ( Chomsky , 1980 ) . This process induces a form of compression , which makes the language simpler and more learnable for the next generation . Linguistic structure , e.g. , compositionality , emerges between these opposing forces . A similar guiding principle can be applied when learning representations of natural data . Many existing methods in self-supervised learning can be considered as attempts to increase expressiveness of the representations , i.e. , by maximizing or controlling the mutual information between inputs and representations , either through reconstruction ( Vincent et al. , 2008 ; Higgins et al. , 2017 ) or contrastive learning ( Oord et al. , 2018 ; Bachman et al. , 2019b ) . We argue that in addition to maximizing expressiveness , a good representation should also be learnable , e.g . support generalization from few examples ( Laina et al. , 2020 ) . In this paper , we propose an evaluation framework for self-supervised learning relying on expressiveness and learnability . Compared with previous work ( e.g. , Wang and Isola , 2020 ) , our framework is independent of ( i ) the availability of downstream task labels or predefined data augmentation techniques , and ( ii ) architecture choices and dimensionality of the latent representations , which makes it applicable to all previous SSL methods . We propose to measure expressiveness in terms estimators of the intrinsic dimension ( ID ) of the data representations ( Ansuini et al. , 2019 ) . Taking inspiration from Laina et al . ( 2020 ) , we quantify learnability in terms of the learning speed ( in terms of number of examples ) of a KNN learner trained to predict the K-Means cluster labels for held out representations . We dub this method Cluster Learnability ( CL ) . We show that high ID and CL correlate with high downstream performance for 30 different pretrained checkpoints across different architectures . To further demonstrate the usefulness of our framework , we present a case-study in which we improve upon DeepCluster ( Caron et al. , 2018 ) , which is a self-supervised learning method that iterates between clustering the representations and fitting the cluster labels . Inspired by our CL measure , we refine the cluster labels by executing KNN at the end of each K-means clustering round . The resulting method , KNN-DeepCluster , is able to outperform DeepCluster in both ImageNet and STL10 . When visualizing the training trajectories on the CL-ID plane , we find that our method indeed improves learnability at convergence . The correlation between performance and learnability holds also for checkpoints obtained during training , which suggests the possibility of using our framework to guide the development of new SSL algorithm , without labels at hand . Our contributions are summarized below : • We analyze self-supervised learning through the lens of expressiveness and learnability , and we propose tractable way to estimate them with Intrinsic Dimension ( ID ) and Cluster Learnability ( CL ) . • We carry out a large-scale evaluation on ImageNet and STL10 with 30 state-of-the-art selfsupervised and supervised checkpoints . We show that CL-ID is more correlated with the Top1 accuracy than previous approaches , with a Pearson ’ s r of 0.93 . • We propose a modification to DeepCluster to increase its cluster learnability . The resulting algorithm , KNN-DeepCluster is able to outperform DeepCluster . 2 RELATED WORK . Representation Evaluation Several recent works address the question of representation evaluation in self-supervised learning . Whitney et al . ( 2021 ) propose to use the learning dynamic of the downstream classifiers to measure the representation complexity , however the method depends on extra human labels . Wang and Isola ( 2020 ) decompose the contrastive loss into an alignment loss and a uniformity loss , which can be used to analyze the representation . However , their method requires access to the projection head output , limiting their applicability to general architecture . Pretext tasks like jigsaw , rotation prediction are shown to be well correlated with the supervised evaluation ( Reed et al. , 2021 ; Deng and Zheng , 2021 ) or architecture search ( Liu et al. , 2020 ) . However , as is shown in our experiments , they only provide limited predictive power when applying to a large pretrained family of checkpoints including both supervised and self-supervised models . In addition , these methods depends on predefined data augmentation , which might be task specific . Manifold Intrinsic Dimension Intrinsic dimension can be thought of as the smallest number of variable needed to approximate the representation manifold . Note that it is different from the intrinsic dimension of loss landscape ( Li et al. , 2018 ) , which measures minimum dimension of parameter space needed to optimize . We build upon extensive work on estimating intrinsic dimension . Applying local neighborhood information to estimate the intrinsic dimension is not a new idea , and it is shown to be more superior to the global eigenvalue approach ( Pettis et al. , 1979b ) . Ansuini et al . ( 2019 ) apply the TwoNN estimator ( Facco et al. , 2017 ) to the non-linear representation manifold of modern deep neural nets . They find that the intrinsic dimension is inversely correlated to the classification accuracy , and that the deep layers have smaller intrinsic dimensions . Their work is further extended to confirm that natural images lies in a low-dimension manifold ( Pope et al. , 2021 ) , and lower ID datasets leads to better generalization . Our work presents a more nuanced view about ID . It suggests that in order to learning representation from scratch , a least amount of ID is needed to convey the information from the dataset , e.g. , a constant-value representation has an ID of 0 , but no capacity to generalize at all . More discussion can be found in Section 5 . Learnability , Ease-of-Transmission and Compression Learnability has been argued to be a hallmark of the human language in order to be effortlessly transmitted through generations ( Kirby et al. , 2014 ; Rafferty et al. , 2011 ; Beckner et al. , 2017 ; Zhou and Yurovsky , 2021 ; Kampen , 2004 ) , and it is also true for visual concepts like color ( Xu et al. , 2010 ) , categories ( Griffiths et al. , 2006 ) , shapes ( Portelance et al. , 2021 ) etc . In deep learning , it has been explored in the context of emergent communication ( Ren et al. , 2020 ; Guo et al. , 2019 ; Li and Bowling , 2019 ) , language drift ( Lu et al. , 2020 ) , and neural module networks ( Vani et al. , 2021 ) , but it is less explored for vision representation learning , except for a human study on just two SSL methods Laina et al . ( 2020 ) . Learnability has a tight connection to compression ( Chaitin , 2007 ) and prequential codelength ( Dawid , 1984 ) , which quantifies the compression levels with the online learning error . Existing works ( Blier and Ollivier , 2018 ) has use it to support the generalization ability of the learner ( e.g. , deep neural nets ) on the dataset ( e.g. , labeled images ) . However , we use it to quantify the learnability of the representation , in the sense that if the emerged Kmeans clustering is more learnable , then the same KNN learner could achieve a lower compression bound via prequential coding . 3 PROPOSED FRAMEWORK . In this section , we introduce our setup and define our measures of expressiveness and learnability . 3.1 NOTATION . We consider the following setting : we assume we have a dataset of images { xi } Ni=1 represented in a pixel space X ⊂ Rd , where xi ∼ P are sampled i.i.d from some natural image distribution P over X . We also consider representation mapsF : X → Rm , typically pretrained neural networks , which represent any image as an m-dimensional vector . Any such map forms a representation dataset Z = { zi } Ni=1 where zi = F ( xi ) . Given zi ∈ Z and an integer k ≥ 1 , we denote by NN ( zi , k ) the k-th nearest neighbor of zi . While we generally mean ‘ nearest ’ with respect to Euclidean distance , in practice we will also use the cosine distance function,1 D ( z1 , z2 ) = 2− 2 cos ( z1 , z2 ) ( 1 ) Let rik = D ( zi , NN ( zi , k ) ) be the distance of the data point zi to its k-th nearest neighbor . 3.2 INTRINSIC DIMENSION ( ID ) . To measure expressiveness , we propose to use the notion of intrinsic dimensionality ( ID ) of the data in the representation space ( Pettis et al. , 1979a ) . Intuitively , as the representation becomes more expressive , we expect it to be able to represent images coming from more fine-grained categories , which could have multiple disentangled features such as color , shape , positions in space , etc . If these disentangled features correspond to different directions in some non-linear manifold , ID should yield a good surrogate for the expressiveness of the representation . Inferring the intrinsic dimension of a highly nonlinear manifold is a challenging problem ( e.g. , Levina and Bickel , 2005 ) . In this work , we leverage the nearest neighbor-based method of Facco et al . ( 2017 ) to estimate ID , which uses only the first two nearest neighbors of each point . This estimator ( TwoNN ) is shown to be reliable with respect to representation dimensions and scalable to real-world datasets with deep neural networks ( Ansuini et al. , 2019 ) . Assuming the data density is approximately constant around data points up to the distance of their neighbors , this method exploits the simple relationship between the distribution of the ratio of first and second nearest neighbor distances and the intrinsic dimension . Formally , let µi = ri2/ri1 be the ratio of distances for the data point zi . We suppose the points are sampled on a manifold with intrinsic dimension d. It can be shown that , under the above local uniformity assumption , µi follows a Pareto distribution with parameter d+ 1 on [ 1 , ∞ ) . That is P ( µ1 , · · · , µN |d ) = dN N∏ i=1 µ − ( d+1 ) i ( 2 ) While d can be computed by maximizing the likelihood , we follow a much simpler method proposed by Ansuini et al . ( 2019 ) and estimate d with a linear regression on the empirical cumulate of the distribution of µ , which is shown to have good practical performance ( Facco et al. , 2017 ) . This estimator is exact for uniformly distributed data . It is also shown to remain asymptotically correct for non-uniform sampling of zi when the image dataset is large enough ( Ansuini et al. , 2019 ) . 1Both distances produce similar results in our experiments .

Summary: This paper tries to give a measurement method to evaluate the learned model, which is highly correlated with the final test accuracy. The measurement method depends on two key factors: Intrinsic Dimension (ID) and CLuster Learnability (CL). This paper claims that the model with higher ID and CL performs better. Using ID and CL to predict top-1 accuracy can achieve a Pearson correlation coefficient of 0.93, which is better than existing predictors (e.g., alignment and uniformity). Inspired by the above observation, this paper proposes a modified DeepCluster algorithm to increase the final performance.