Contrastive Representation Distillation

1 INTRODUCTION . Knowledge distillation ( KD ) transfers knowledge from one deep learning model ( the teacher ) to another ( the student ) . The objective originally proposed by Hinton et al . ( 2015 ) minimizes the KL divergence between the teacher and student outputs . This formulation makes intuitive sense when the output is a distribution , e.g. , a probability mass function over classes . However , often we instead wish to transfer knowledge about a representation . For example , in the problem of “ cross-modal distillation '' , we may wish to transfer the representation of an image processing network to a sound ( Aytar et al. , 2016 ) or to depth ( Gupta et al. , 2016 ) processing network , such that deep features for an image and the associated sound or depth features are highly correlated . In such cases , the KL divergence is undefined . Representational knowledge is structured – the dimensions exhibit complex interdependencies . The original KD objective introduced in ( Hinton et al. , 2015 ) treats all dimensions as independent , conditioned on the input . Let yT be the output of the teacher and yS be the output of the student . Then the original KD objective function , ψ , has the fully factored form : ψ ( yS , yT ) = ∑ i φi ( y S i , y T i ) ∗ . Such a factored objective is insufficient for transferring structural knowledge , i.e . dependencies between output dimensions i and j . This is similar to the situation in image generation where an L2 objective produces blurry results , due to independence assumptions between output dimensions . To overcome this problem , we would like an objective that captures correlations and higher order output dependencies . To achieve this , in this paper we leverage the family of contrastive objectives ( Gutmann & Hyvärinen , 2010 ; Oord et al. , 2018 ; Arora et al. , 2019 ; Hjelm et al. , 2018 ) . These objective functions have been used successfully in recent years for density estimation and representation learning , especially in self-supervised settings . Here we adapt them to the task of knowledge distillation from one deep network to another . We show that it is important to work in representation space , similar to recent works such as Zagoruyko & Komodakis ( 2016a ) ; Romero et al . ( 2014 ) . However , note that the loss functions used in those works do not explicitly try to capture correlations or higher-order dependencies in representational space . Code : http : //github.com/HobbitLong/RepDistiller . ∗In particular , in ( Hinton et al. , 2015 ) , φi ( a , b ) = −a log b ∀i Our objective maximizes a lower-bound to the mutual information between the teacher and student representations . We find that this results in better performance on several knowledge transfer tasks . We conjecture that this is because the contrastive objective better transfers all the information in the teacher ’ s representation , rather than only transferring knowledge about conditionally independent output class probabilities . Somewhat surprisingly , the contrastive objective even improves results on the originally proposed task of distilling knowledge about class probabilities , for example , compressing a large CIFAR100 network into a smaller one that performs almost as well . We believe this is because the correlations between different class probabilities contains useful information that regularizes the learning problem . Our paper forges a connection between two literatures that have evolved mostly independently : knowledge distillation and representation learning . This connection allows us to leverage strong methods from representation learning to significantly improve the SOTA on knowledge distillation . Our contributions are : 1 . A contrastive-based objective for transferring knowledge between deep networks . 2 . Applications to model compression , cross-modal transfer , and ensemble distillation . 3 . Benchmarking 12 recent distillation methods ; CRD outperforms all other methods , e.g. , 57 % average relative improvement over the original KD ( Hinton et al. , 2015 ) † , which , surprisingly , performs the second best . 2 RELATED WORK . The seminal work of Buciluǎ et al . ( 2006 ) and Hinton et al . ( 2015 ) introduced the idea of knowledge distillation between large , cumbersome models into smaller , faster models without losing too much generalization power . The general motivation was that at training time , the availability of computation allows “ slop '' in model size , and potentially faster learning . But computation and memory constraints at inference time necessitate the use of smaller models . Buciluǎ et al . ( 2006 ) achieve this by matching output logits ; Hinton et al . ( 2015 ) introduced the idea of temperature in the softmax outputs to better represent smaller probabilities in the output of a single sample . These smaller probabilities provide useful information about the learned representation of the teacher model ; some tradeoff between large temperatures ( which increase entropy ) or small temperatures tend to provide the highest transfer of knowledge between student and teacher . The method in ( Li et al. , 2014 ) was also closely related to ( Hinton et al. , 2015 ) . Attention transfer ( Zagoruyko & Komodakis , 2016a ) focuses on the features maps of the network as opposed to the output logits . Here the idea is to elicit similar response patterns in the teacher and student feature maps ( called “ attention '' ) . However , only feature maps with the same spatial †Average relative improvement = 1 N ∑N i=1 Accicrd−Acc i kd Acci kd −Accivan , where Accicrd , Acc i kd , and Acc i van represent the accuracies of CRD , KD , and vanilla training of the i-th student model , respectively . resolution can be combined in this approach , which is a significant limitation since it requires student and teacher networks with very similar architectures . This technique achieves state of the art results for distillation ( as measured by the generalization of the student network ) . FitNets ( Romero et al. , 2014 ) also deal with intermediate representations by using regressions to guide the feature activations of the student network . Since Zagoruyko & Komodakis ( 2016a ) do a weighted form of this regression , they tend to perform better . Other papers ( Yim et al. , 2017 ; Huang & Wang , 2017 ; Kim et al. , 2018 ; Yim et al. , 2017 ; Huang & Wang , 2017 ; Ahn et al. , 2019 ; Koratana et al. , 2019 ) have enforced various criteria based on representations . The contrastive objective we use in this paper is the same as that used in CMC ( Tian et al. , 2019 ) . But we derive from a different perspective and give a rigorous proof that our objective is a lower bound on mutual information . Our objective is also related to the InfoNCE and NCE objectives introduced in ( Oord et al. , 2018 ; Gutmann & Hyvärinen , 2010 ) . Oord et al . ( 2018 ) use contrastive learning in the context of self-supervised representations learning . They show that their objective maximizes a lower bound on mutual information . A very related approach is used in ( Hjelm et al. , 2018 ) . InfoNCE and NCE are closely related but distinct from adversarial learning ( Goodfellow et al. , 2014 ) . In ( Goodfellow , 2014 ) , it is shown that the NCE objective of Gutmann & Hyvärinen ( 2010 ) can lead to maximum likelihood learning , but not the adversarial objective . 3 METHOD . The key idea of contrastive learning is very general : learn a representation that is close in some metric space for “ positive ” pairs and push apart the representation between “ negative ” pairs . Fig . 1 gives a visual explanation for how we structure contrastive learning for the three tasks we consider : model compression , cross-modal transfer and ensemble distillation . 3.1 CONTRASTIVE LOSS . Given two deep neural networks , a teacher fT and a student fS . Let x be the network input ; we denote representations at the penultimate layer ( before logits ) as fT ( x ) and fS ( x ) respectively . Let xi represent a training sample , and xj another randomly chosen sample . We would like to push closer the representations fS ( xi ) and fT ( xi ) while pushing apart fS ( xi ) and fT ( xj ) . For ease of notation , we define random variables S and T for the student and teacher ’ s representations of the data respectively : x ∼ pdata ( x ) / data ( 1 ) S = fS ( x ) / student ’ s representation ( 2 ) T = fT ( x ) / teacher ’ s representation ( 3 ) Intuitively speaking , we will consider the joint distribution p ( S , T ) and the product of marginal distributions p ( S ) p ( T ) , so that , by maximizing KL divergence between these distributions , we can maximize the mutual information between student and teacher representations . To setup an appropriate loss that can achieve this aim , let us define a distribution q with latent variable C which decides whether a tuple ( fT ( xi ) , fS ( xj ) ) was drawn from the joint ( C = 1 ) or product of marginals ( C = 0 ) : q ( T , S|C = 1 ) = p ( T , S ) , q ( T , S|C = 0 ) = p ( T ) p ( S ) ( 4 ) Now , suppose in our data , we are given 1 congruent pair ( drawn from the joint distribution , i.e . the same input provided to T and S ) for every N incongruent pairs ( drawn from the product of marginals ; independent randomly drawn inputs provided to T and S ) . Then the priors on the latent C are : q ( C = 1 ) = 1 N + 1 , q ( C = 0 ) = N N + 1 ( 5 ) By simple manipulation and Bayes ’ rule , the posterior for class C = 1 is given by : q ( C = 1|T , S ) = q ( T , S|C = 1 ) q ( C = 1 ) q ( T , S|C = 0 ) q ( C = 0 ) + q ( x , y|C = 1 ) q ( C = 1 ) ( 6 ) = p ( T , S ) p ( T , S ) +Np ( T ) p ( S ) ( 7 ) Next , we observe a connection to mutual information as follows : log q ( C = 1|T , S ) = log p ( T , S ) p ( T , S ) +Np ( T ) p ( S ) = − log ( 1 +N p ( T ) p ( S ) p ( T , S ) ) ≤ − log ( N ) + log p ( T , S ) p ( T ) p ( S ) ( 8 ) Then taking expectation on both sides w.r.t . p ( T , S ) ( equivalently w.r.t . q ( T , S|C = 1 ) ) and rearranging , gives us : I ( T ; S ) ≥ log ( N ) + Eq ( T , S|C=1 ) log q ( C = 1|T , S ) / MI bound ( 9 ) where I ( T ; S ) is the mutual information between the distributions of the teacher and student embeddings . Thus maximizing Eq ( T , S|C=1 ) log q ( C = 1|T , S ) w.r.t . the parameters of the student network S increases a lower bound on mutual information . However , we do not know the true distribution q ( C = 1|T , S ) ; instead we estimate it by fitting a model h : { T , S } → [ 0 , 1 ] to samples from the data distribution q ( C = 1|T , S ) , where T and S represent the domains of the embeddings . We maximize the log likelihood of the data under this model ( a binary classification problem ) : Lcritic ( h ) = Eq ( T , S|C=1 ) [ log h ( T , S ) ] +NEq ( T , S|C=0 ) [ 1− log ( h ( T , S ) ) ] ( 10 ) h∗ = argmax h Lcritic ( h ) / optimal critic ( 11 ) We term h the critic since we will be learning representations that optimize the critic ’ s score . Assuming sufficiently expressive h , h∗ ( T , S ) = q ( C = 1|T , S ) ( via Gibbs ’ inequality ; see Sec . 6.2.1 for proof ) , so we can rewrite Eq . 9 in terms of h∗ : I ( T ; S ) ≥ log ( N ) + Eq ( T , S|C=1 ) [ log h∗ ( T , S ) ] ( 12 ) Therefore , we see that the optimal critic is an estimator whose expectation lower-bounds mutual information . We wish to learn a student that maximizes the mutual information between its representation and the teacher ’ s , suggesting the following optimization problem : fS∗ = argmax fS Eq ( T , S|C=1 ) [ log h∗ ( T , S ) ] ( 13 ) An apparent difficulty here is that the optimal critic h∗ depends on the current student . We can circumvent this difficulty by weakening the bound in ( 12 ) to : I ( T ; S ) ≥ log ( N ) + Eq ( T , S|C=1 ) [ log h∗ ( T , S ) ] +NEq ( T , S|C=0 ) [ log ( 1− h∗ ( T , S ) ) ] ( 14 ) = log ( N ) + Lcritic ( h∗ ) = log ( N ) + max h Lcritic ( h ) ( 15 ) ≥ log ( N ) + Lcritic ( h ) ( 16 ) The first line comes about by simply adding NEq ( T , S|C=0 ) [ log ( 1− h∗ ( T , S ) ) ] to the bound in ( 12 ) . This term is strictly negative , so the inequality holds . The last line follows from the fact that Lcritic ( h∗ ) upper-bounds Lcritic ( h ) . Optimizing ( 15 ) w.r.t . the student we have : fS∗ = argmax fS max h Lcritic ( h ) / our final learning problem ( 17 ) = argmax fS max h Eq ( T , S|C=1 ) [ log h ( T , S ) ] +NEq ( T , S|C=0 ) [ log ( 1− h ( T , S ) ) ] ( 18 ) which demonstrates that we may jointly optimize fS at the same time as we learn h. We note that due to ( 16 ) , fS∗ = argmaxfS Lcritic ( h ) , for any h , also is a representation that optimizes a lower-bound ( a weaker one ) on mutual information , so our formulation does not rely on h being optimized perfectly . We may choose to represent h with any family of functions that satisfy h : { T , S } → [ 0 , 1 ] . In practice , we use the following : h ( T , S ) = eg T ( T ) ′gS ( S ) /τ egT ( T ) ′gS ( S ) /τ + NM ( 19 ) where M is the cardinality of the dataset and τ is a temperature that adjusts the concentration level . In practice , since the dimensionality of S and T may be different , gS and gT linearly transform them into the same dimension and further normalize them by L-2 norm before the inner product . The form of Eq . ( 18 ) is inspired by NCE ( Gutmann & Hyvärinen , 2010 ; Wu et al. , 2018 ) . Our formulation is similar to the InfoNCE loss ( Oord et al. , 2018 ) in that we maximize a lower bound on the mutual information . However we use a different objective and bound , which in our experiments we found to be more effective than InfoNCE . Implementation . Theoretically , larger N in Eq . 16 leads to tighter lower bound on MI . In practice , to avoid using very large batch size , we follow Wu et al . ( 2018 ) and implement a memory buffer that stores latent features of each data sample computed from previous batches . Therefore , during training we can efficiently retrieve a large number of negative samples from the memory buffer .

Summary: This paper combines a contrastive objective measuring the mutual information between the representations learned by a teacher and a student networks for model distillation. The objective enforces correlations between the learned representations. When combined with the popular KL divergence between the predictions of the two networks, the proposed model shows consistently improvement over existing alternatives on three distillation tasks.