DHOG: Deep Hierarchical Object Grouping

1 INTRODUCTION . It is very expensive to label a dataset with respect to a particular task . Consider the alternative where a user , instead of labelling a dataset , specifies a simple set of class-preserving transformations or ‘ augmentations ’ . For example , lighting changes will not change a dog into a cat . Is it possible to learn a model that produces a useful representation by leveraging a set of such augmentations ? This representation would need to be good at capturing salient information about the data , and enable downstream tasks to be done efficiently . If the representation were a discrete labelling which groups the dataset into clusters , an obvious choice of downstream task is unsupervised clustering . Ideally the clusters should match direct labelling , without ever having been learnt on explicitly labelled data . Using data augmentations to drive unsupervised representation learning for images has been explored by a number of authors ( Dosovitskiy et al. , 2014 ; 2015 ; Bachman et al. , 2019 ; Chang et al. , 2017 ; Wu et al. , 2019 ; Ji et al. , 2019 ; Cubuk et al. , 2019 ) . These approaches typically involve learning neural networks that map augmentations of the same image to similar representations , which is reasonable since variances across many common augmentations often align with the invariances we would require . A number of earlier works target maximising mutual information ( MI ) between augmentations ( van den Oord et al. , 2018 ; Hjelm et al. , 2019 ; Wu et al. , 2019 ; Ji et al. , 2019 ; Bachman et al. , 2019 ) . Targetting high MI between representations computed from distinct augmentations enables learning representations that capture the invariances induced by the augmentations . We are interested in a particularly parsimonious representation : a discrete labelling of the data . This labelling can be seen as a clustering ( Ji et al. , 2019 ) procedure , where MI can be computed and assessment can be done directly using the learned labelling , as opposed to via an auxiliary network trained posthoc . 1.1 SUBOPTIMAL MUTUAL INFORMATION MAXIMISATION . We argue and show that the MI objective is not maximised effectively in existing work due to the combination of : 1 . Greedy optimisation algorithms used to train neural networks , such as stochastic gradient descent ( SGD ) that potentially target local optima ; and 2 . A limited set of data augmentations that can result in the existence of multiple local optima to the MI maximisation objective . SGD is greedy in the sense that early-found high-gradient features can dominate and so networks will tend to learn easier-to-compute locally-optimal representations ( for example , one that can be computed using fewer neural network layers ) over those that depend on complex features . By way of example , in natural images , average colour is an easy-to-compute characteristic , whereas object type is not . If the augmentation strategy preserves average colour , then a reasonable mapping need only compute colour information , and high MI between learned image representations will be obtained . This result is suboptimal in the sense that a hypothetical higher MI optima exists that also captures semantic information , assuming the model has sufficient capacity to learn and represent this . The conceivable existence of many such local optima coupled with greedy optimisation presents a challenge : how can we leverage powerful image augmentation-driven MI objectives while avoiding greedily-found local optima ? Dealing with greedy solutions Heuristic solutions , such as as Sobel edge-detection ( Caron et al. , 2018 ; Ji et al. , 2019 ) as a pre-processing step , have been suggested to remove/alter the features in images that may cause trivial representations to be learned . This is a symptomatic treatment and not a solution . In the work presented herein , we acknowledge that greedy SGD can get stuck in local optima of the MI maximisation objective because of limited data augmentations . Instead of trying to prevent a greedy solution , our technique lets a model learn this representation , but also requires it to learn an additional distinct representation . Specifically , we minimise the MI between these two representations so that the latter can not rely on the same features . We extend this idea by adding representations , each time requiring the latest to be distinct from all previous representations . Downstream task : clustering For this work , our focus is on finding higher MI representations ; we then assess the downstream capability on the ground truth task of image classification , meaning that we can either ( 1 ) learn a representation that must be ‘ decoded ’ via an additional learning step , or ( 2 ) produce a discrete labelling that requires no additional learning . Clustering methods offer a direct comparison and require no labels for learning a mapping from the learned representation to class labels . Instead , labels are only required to assign groups to appropriate classes and no learning is done using these . Our comparisons are with respect to clustering methods . 1.2 CONTRIBUTIONS . Learning a set of representations by encouraging them to have low MI , while still maximising the original augmentation-driven MI objective for each representation , is the core idea behind Deep Hierarchical Object Grouping ( DHOG ) . We define a mechanism to produce a set of hierarchicallyordered solutions ( in the sense of easy-to-hard orderings , not tree structures ) . DHOG is able to better maximise the original MI objective between augmentations since each representation must correspond to a unique local optima . Our contributions are : 1 . We demonstrate that current methods do not effectively maximise the MI objective1 because greedy stochastic gradient descent ( SGD ) typically results in suboptimal local optima . To mitigate for this problem , we introducing DHOG : a robust neural network image grouping method to learn diverse and hierarchically arranged sets of discrete image labellings ( Section 3 ) by explicitly modelling , accounting for , and avoiding spurious local optima , requiring only simple data augmentations , and needing no Sobel edge detection . 2 . We show a marked improvement over the current state-of-the-art for standard benchmarks in end-to-end image clustering for CIFAR-10 , CIFAR-100-20 ( a 20-way class grouping of CIFAR-100 , and SVHN ; we set a new accuracy benchmarks on CINIC-10 ; and show the utility of our method on STL-10 ( Section 4 ) . To be clear , DHOG still learns to map data augmentations to similar representations as this is imperative to the learning process . The difference is that DHOG enables a number of intentionally distinct data labellings to be learned , arranged hierarchically in terms of source feature complexity . 1We show this by finding higher mutual information solutions using DHOG , rather than by any analysis of the solutions themselves . 2 RELATED WORK . The idea of MI maximisation for representation learning is called the infoMAX principle ( Linsker , 1988 ; Tschannen et al. , 2019 ) . Contrastive predictive coding ( van den Oord et al. , 2018 ) ( CPC ) models a 2D latent space using an autoregressive model and defines a predictive setup to maximise MI between distinct spatial locations . Deep InfoMAX ( Hjelm et al. , 2019 ) ( DIM ) does not maximise MI across a set of data augmentations , but instead uses mutual information neural estimation ( Belghazi et al. , 2018 ) and negative sampling to balance maximising MI between global representations and local representations . Augmented multiscale Deep InfoMAX ( Bachman et al. , 2019 ) ( AMDIM ) incorporates MI maximisation across data augmentations and multiscale comparisons . Clustering approaches are more directly applicable for comparison with DHOG because they explicitly learn a discrete labelling . The authors of deep embedding for clustering ( DEC ) ( Xie et al. , 2016 ) focused their attention on jointly learning an embedding suited to clustering and a clustering itself . They argued that the notion of distance in the feature space is crucial to a clustering objective . Joint unsupervised learning of deep representations and image clusters ( JULE ) ( Yang et al. , 2016 ) provided supervisory signal for representation learning . Some methods ( Ghasedi Dizaji et al. , 2017 ; Fard et al. , 2018 ) employ autoencoder architectures along with careful regularisation of cluster assignments to ( 1 ) ensure sufficient information retention , and ( 2 ) avoid cluster degeneracy ( i.e. , mapping all images to the same class ) . Deep adaptive clustering ( Chang et al. , 2017 ) ( DAC ) recasts the clustering problem as binary pairwise classification , pre-selecting comparison samples via feature cosine distances . A constraint on the DAC system allows for a one-hot encoding that avoids cluster degeneracy . Another mechanism for dealing with degeneracy is to use a standard clustering algorithm , such asK-means to iteratively group on learned features . This approach is used by DeepCluster ( Caron et al. , 2018 ) . Associative deep clustering ( ADC ) ( Haeusser et al. , 2018 ) uses the idea that associations in the embedding space are useful for learning . A network was learned to associate data with ( pseudolabelled ) centroids . They leveraged augmentations by encouraging samples to output similar cluster probabilities . Deep comprehensive correlation mining ( Wu et al. , 2019 ) ( DCCM ) constructs a sample correlation graph for pseudo-labels and maximises the MI between augmentations , and the MI between local and global features for each augmentation . While many of the aforementioned methods estimate MI in some manner , invariant information clustering ( Ji et al. , 2019 ) ( IIC ) directly defines the MI using the c-way softmax output ( i.e. , probability of belong to class c ) , and maximises this over data augmentations to learn clusters . They effectively avoid degenerate solutions because MI maximisation implicitly targets marginal entropy . We use the same formulation for MI in Section 3 . 3 METHOD . Figure 1 shows the DHOG architecture . DHOG is an approach for obtaining jointly trained multilevel representations as discrete labellings , arranged in a simple-to-complex hierarchy , and computed by separate ‘ heads ’ . A head is an unit that computes a multivariate class probability vector . By requiring low MI between heads , a diversity of solutions to the MI maximisation objective can be found . The head that best maximises MI between augmentations typically aligns better with a ground truth task that also relies on complex features that augmentations are designed to preserve . Figure 1 demonstrates the DHOG architecture and training principles . There are shared model weights ( 2 : ResNet blocks 1 , 2 , and 3 ) and head-specific weights ( the MLP layers and 3 : ResNet blocks 4 to 8 ) . For the sake of brevity , we abuse notation and use MI ( z , z′ ) between labelling probability vectors as an overloaded shorthand for the mutual information MI ( c , c′ ) between the labelling random variables c and c′ that have probability vectors z and z′ respectively . Any branch of the DHOG architecture ( 1 to any zi ) can be regarded as a single neural network . These are trained to maximise the MI between the label variables at each head for different augmentations ; i.e. , between label variables with probability vectors zi ( x ) and zi ( x′ ) for augmentations x and x′ . Four augmentations are shown at 1 . The MI is maximised pairwise between all pairs , at 4 . This process can be considered pulling the mapped representations together . block is repeated k − 3 times ( k = 8 here ) . 1 Augmentations of each image , xa ... d , are separately processed by the network . 2 Each shallow ResNet block ( 1 . . . 3 ) constitutes shared computation for deeper blocks , while also computing separate probability vectors , z1 . . . z3 . Each zi is viewed as the probability for each outcome of the random variable ci that makes a discrete labelling choice . 3 The deepest ResNet blocks compute further z > 3 . 4 The network is trained by maximising the MI between allocations ci from all data augmentations , and 5 separately for each node i , minimising the MI between ci and c < i for the same data augmentation . 6 This is implemented by stopping gradients such that they are not back-propagated for later computation paths ( red crosses ) . Following IIC ( Ji et al. , 2019 ) , we compute the MI directly from the label probability vectors within a minibatch . Let zi , z′i denote the random probability vectors at head i associated with sampling a data item and its augmentations , and passing those through the network . Then we can compute the mutual MI between labels associated with each augmentation using MIaug ( ci , c ′ i ) = Tr ( E [ zi ( z ′ i ) T ] T log ( E [ zi ( z ′ i ) T ] ) ) − E [ zTi ] logE [ zi ] − E [ ( z′i ) T ] logE [ ( z′i ) ] , ( 1 ) where Tr is the matrix trace , logarithms are computed element-wise , and expectations are over data samples and augmentations of each sample . In practice we compute an empirical estimate of this MI based on samples from a minibatch .

Summary: This paper addresses the problem of unsupervised learning of class representation using data augmentation. Its key idea is to encourage the learned representations to have low MI while maximizing the original augmentation-driven MI objective. It reports the improved performance for the benchmarks of Ji et al. 2019 – classification on some easy datasets (e.g. CIFAR-10, CINIC-10, SVHN and STL-10).