CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP

1 INTRODUCTION . With the advent of large corpora of unlabeled data in vision and language , self-supervised learning via contrastive learning has become highly successful . Some contrastive learning objectives , such as those of BYOL ( Grill et al. , 2020 ) and SimSiam ( Chen & He , 2021 ) , do not require negative samples . However , the most popular objective for contrastive learning is InfoNCE ( van den Oord et al. , 2018 ) , in which for an anchor sample , a positive sample is contrasted with negative samples . The idea to use objectives with negative samples is well known in deep learning ( Gutmann & Hyvärinen , 2010 ; Chen et al. , 2017 ; Mikolov et al. , 2013 ) . For contrastive learning , the most successful objective is InfoNCE , which has been introduced as Contrastive Predictive Coding ( CPC ) ( van den Oord et al. , 2018 ) . InfoNCE has been applied to transfer learning ( Hénaff et al. , 2019 ) , to natural language response suggestion ( Henderson et al. , 2017 ) , to learning sentence representations from unlabelled data ( Logeswaran & Lee , 2018 ) , and to unsupervised feature learning by maximizing distinctions between instances ( Wu et al. , 2018 ) . InfoNCE has been used for learning visual representations in Pretext-Invariant Representation Learning ( PIRL ) ( Misra & vanDerMaaten , 2020 ) , in Momentum Contrast ( MoCo ) ( He et al. , 2020 ) , and in SimCLR ( Chen et al. , 2020 ) . SimCLR became well known as is was highly effective for transfer learning . Zero-shot transfer learning ( Lampert et al. , 2009 ) is one of the most ambitious goals in vision , since it would improve various real-world downstream applications . Current models in natural language processing and vision perform very well on standard benchmarks , but they fail at new data , new applications , deployments in the wild , and stress tests ( D ’ Amour et al. , 2020 ; Recht et al. , 2019 ; Taori et al. , 2020 ; Lapuschkin et al. , 2019 ; Geirhos et al. , 2020 ) . A model with high zero-shot transfer learning performance will not fail on such data , therefore will be trusted by practitioners . Contrastive Language-Image Pre-training ( CLIP ) based on the InfoNCE objective yielded very impressive results at zero-shot transfer learning ( Radford et al. , 2021 ) . CLIP learns expressive image embeddings directly from raw text , thereby leverages a much richer source of supervision than just labels . A plethora of CLIP follow-up work has already been published ( see Appendix Section A.5 ) . The CLIP model is considered as an important foundation model ( Bommasani et al. , 2021 ) . Though CLIP excels at zero-shot transfer learning , it can be improved . CLIP training suffers from an “ explaining away ” problem ( Wellman & Henrion , 1993 ) , which leads to “ shortcut learning ” ( Geirhos et al. , 2020 ) or the Clever Hans phenomenon ( Lapuschkin et al. , 2019 ) . Explaining away impedes the increase of the similarity between a text and a corresponding image , since learning focuses on only one common aspect and does not exploit the full covariance structure of the data . If one common aspect is sufficient for high similarity , the InfoNCE objective saturates , since it has the form a/ ( a+ b ) with a giving the similarity of a matched pair and b giving the average similarity of unmatched pairs . For a large similarity a , the objective saturates and increasing a has a small effect . Contrary to InfoNCE , the leave-one-out ( “ InfoLOOB ” ) bound ( Poole et al. , 2019 ) is of the form a/b which does not saturate . However , so far the InfoLOOB bound was not used as an objective in contrastive learning . We justify the maximization of the InfoLOOB bound for contrastive learning in Appendix Section A.1.3 . We show that maximizing the InfoLOOB bound leads to a good approximation of the mutual information , in particular for high mutual information . A problem of InfoLOOB is that it has high variance for small b . Even when InfoLOOB avoids saturation , CLIP insufficiently extracts the covariance structure in the data . The covariance originates from co-occurrences of related words in text or from co-occurrences of objects , textures , or colors in images . CLIP ’ s problem of insufficiently extracting the covariance structure of the data is tackled by modern Hopfield networks . Hopfield networks are energy-based , binary associative memories , which popularized artificial neural networks in the 1980s ( Hopfield , 1982 ; 1984 ) . Associative memory networks have been designed to store and retrieve samples . Their storage capacity can be considerably increased by polynomial terms in the energy function ( Chen et al. , 1986 ; Psaltis & Cheol , 1986 ; Baldi & Venkatesh , 1987 ; Gardner , 1987 ; Abbott & Arian , 1987 ; Horn & Usher , 1988 ; Caputo & Niemann , 2002 ; Krotov & Hopfield , 2016 ) . In contrast to these binary memory networks , we use continuous associative memory networks with very high storage capacity . These modern Hopfield networks for deep learning architectures have an energy function with continuous states and can retrieve samples with only one update ( Ramsauer et al. , 2021 ; 2020 ) . Modern Hopfield Networks have already been successfully applied to immune repertoire classification ( Widrich et al. , 2020 ) and chemical reaction prediction ( Seidl et al. , 2021 ) . Modern Hopfield networks reinforce the covariance structure in the data and stabilize the InfoLOOB objective by increasing b . The covariance structure of retrieved embeddings is amplified through co-occurrences of embedding features in the memory . Additionally , the retrieved embeddings are less noisy and more similar to one another which leads to a larger b . We introduce “ Contrastive Leave One Out Boost ” ( CLOOB ) which overcomes CLIP ’ s problems of ( i ) “ explaining away ” with saturation and ( ii ) insufficiently extracting the covariance structure of the data . CLOOB uses the leave-one-out ( “ InfoLOOB ” ) bound ( Poole et al. , 2019 ) as the objective in combination with modern Hopfield networks . Our contributions are : ( a ) we introduce a new contrastive learning method called CLOOB , ( b ) we propose InfoLOOB as an objective for contrastive learning , ( c ) we propose to use modern Hopfield networks to reinforce covariance structures , ( d ) we show theoretical properties of the InfoLOOB objective and loss function . 2 INFOLOOB VS. INFONCE . We discuss and analyse known bounds on the mutual information I ( X ; Y ) between random variables X and Y , which are distributed according to p ( x , y ) : I ( X ; Y ) = Ep ( x , y ) [ ln p ( x , y ) p ( x ) p ( y ) ] = Ep ( x , y ) [ ln p ( x | y ) p ( x ) ] = Ep ( x , y ) [ ln p ( y | x ) p ( y ) ] . ( 1 ) We consider the multi-sample lower bound “ InfoNCE ” ( van den Oord et al. , 2018 ) . A pair of an anchor sample y and a positive sample x1 is drawn via the joint distribution p ( x1 , y ) . The negative samples X̃ = { x2 , . . . , xN } are drawn iid according to the marginal distribution p ( x ) . Using X = { x1 , x2 , . . . , xN } , the probabilities of the datasets are p ( X̃ ) = ∏N i=2 p ( xi ) , p ( X | y ) = p ( x1 | y ) ∏N i=2 p ( xi ) , and p ( X ) = ∏N i=1 p ( xi ) . The InfoNCE with score function f ( x , y ) is IInfoNCE ( X1 ; Y ) = Ep ( y ) [ Ep ( X|y ) [ ln ( f ( x1 , y ) 1 N ∑N i=1 f ( xi , y ) ) ] ] , ( 2 ) using the factor 1/N as in Poole et al . ( 2019 ) ; Tschannen et al . ( 2019 ) ; Cheng et al . ( 2020 ) ; Chen et al . ( 2021 ) . For f ( x , y ) = p ( y | x ) , we obtain the InfoNCE with probabilities . The InfoNCE is a lower bound on the mutual information ( Poole et al. , 2019 ) , which is stated in the next theorem . Theorem 1 ( InfoNCE lower bound ) . InfoNCE with score function f ( x , y ) is a lower bound on the mutual information : I ( X1 ; Y ) ≥ Ep ( y ) [ Ep ( X|y ) [ ln ( f ( x1 , y ) 1 N ∑N i=1 f ( xi , y ) ) ] ] = IInfoNCE ( X1 ; Y ) . ( 3 ) In particular , the bound holds for InfoNCE with probabilities , i.e . for f ( x , y ) = p ( y | x ) . For a proof see Poole et al . ( 2019 ) and the proof of Theorem A1 in the Appendix . The “ Leave one out upper bound ” ( Poole et al. , 2019 ) on the mutual information was called “ L1Out ” in Cheng et al . ( 2020 ) , while we call it “ InfoLOOB ” ( LOOB for “ Leave One Out Bound ” ) . InfoLOOB is the same as InfoNCE ( Eq . ( 3 ) ) , but without the positive sample x1 in the denominator . Contrastive Log-ratio Upper Bound ( CLUB ) , another upper bound on the mutual information , was only used for minimizing it ( Cheng et al. , 2020 ) . Maximizing CLUB failed in experiments , because the embedding distribution was not uniform as known for similar objectives ( Wang & Liu , 2021 ) . Uniform embedding distributions are required for successful contrastive learning ( Wang & Isola , 2020 ) . We use InfoLOOB as an objective , since it approximates high mutual information better than InfoNCE . Maximizing an upper bound on the mutual information might be counter-intuitive . Therefore , we justify the maximization of the InfoLOOB bound for contrastive learning in Appendix Section A.1.3 . We show that maximizing the InfoLOOB bound approximates the mutual information , the better the higher it is . Recently , InfoLOOB was independently introduced for and successfully applied to image-to-image contrastive learning ( Yeh et al. , 2021 ) . The InfoLOOB with score function f ( x , y ) is defined in the following , where we obtain the InfoLOOB with probabilities for f ( x , y ) = p ( y | x ) : IInfoLOOB ( X1 ; Y ) = Ep ( y ) [ Ep̃ ( X|y ) [ ln ( f ( x1 , y ) 1 N−1 ∑N i=2 f ( xi , y ) ) ] ] . ( 4 ) Before we show that InfoLOOB with a score function is an upper bound on the mutual information , we need some definitions . p̃ ( x | y ) draws the positives for y with lower probability than p ( x ) , that is , the positives are under-sampled . Z ( y ) = Ep̃ ( x|y ) [ f ( x , y ) ] gives the average score f ( x , y ) , if under-sampling via p̃ ( x | y ) , while Z∗ ( y ) = Ep ( x ) [ f ( x , y ) ] average score f ( x , y ) if sampling from p ( x ) . We define the variational distribution q ( x | y ) = p ( x ) f ( x , y ) Z∗ ( y ) . Our main assumption is expressed by the log-ratio of the averages Z ( y ) and Z∗ ( y ) : Ep ( y ) [ KL ( p ( x | y ) ‖ q ( x | y ) ) ] 6 Ep ( y ) [ lnZ∗ ( y ) − lnZ ( y ) ] , ( 5 ) which ensures that the positives x are sufficiently under-sampled via p ( x | y ) . The Kullback-Leibler divergence gives the minimal difference between averaging f ( x , y ) via p ( x ) and via p̃ ( x | y ) . The next theorem shows that InfoLOOB is an upper bound on the mutual information . Theorem 2 ( InfoLOOB upper bound ) . If X̃ = { x2 , . . . , xN } are drawn iid according to p̃ ( x | y ) and if the main assumption Eq . ( 5 ) holds , then InfoLOOB with score function f ( x , y ) is an upper bound on the mutual information : I ( X1 ; Y ) 6 Ep ( y ) [ Ep̃ ( X|y ) [ ln ( f ( x1 , y ) 1 N−1 ∑N i=2 f ( xi , y ) ) ] ] = IInfoLOOB ( X1 ; Y ) . ( 6 ) The bound is valid for InfoLOOB with probabilities ( without under-sampling ) , where the negative samples X̃ = { x2 , . . . , xN } are drawn iid according to p ( x ) and f ( x , y ) = p ( y | x ) . The proof for this theorem is given as proof for Theorem A2 in the Appendix . Loss functions and their gradients . The training set { ( x1 , y1 ) , ( x2 , y2 ) , . . . , ( xN , yN ) } consists of N samples that are drawn iid from p ( x , y ) . InfoNCE uses the matrixX = ( x1 , . . . , xN ) , while InfoLOOB uses X̃ = ( x2 , . . . , xN ) . The matrices differ by the positive sample x1 . For the score function f ( x , y ) , we use f ( x , y ) = exp ( τ−1sim ( x , y ) ) with the similarity sim ( x , y ) = yTx and τ as the temperature . We have the InfoNCE and InfoLOOB loss functions : LInfoNCE = − 1 N N∑ i=1 ln exp ( τ−1 xTi yi ) ∑N j=1 exp ( τ −1 xTi yj ) − 1 N N∑ i=1 ln exp ( τ−1 xTi yi ) ∑N j=1 exp ( τ −1 xTj yi ) , ( 7 ) LInfoLOOB = − 1 N N∑ i=1 ln exp ( τ−1 xTi yi ) ∑N j 6=i exp ( τ −1 xTi yj ) − 1 N N∑ i=1 ln exp ( τ−1 xTi yi ) ∑N j 6=i exp ( τ −1 xTj yi ) . ( 8 ) In the second sum of the losses in Eq . 7 and Eq . 8 , we consider only the first term . For simplicity , we abbreviate y = y1 leading to the pair ( x1 , y ) and the negatives X̃ = ( x2 , . . . , xN ) . LInfoNCE ( y ) = − ln exp ( τ−1 xT1 y ) ∑N j=1 exp ( τ −1 xTj y ) , LInfoLOOB ( y ) = − ln exp ( τ−1 xT1 y ) ∑N j=2 exp ( τ −1 xTj y ) . These loss terms can be simplified to LInfoNCE ( y ) = −τ−1yTx1 + τ−1lse ( τ−1 , XTy ) and LInfoLOOB ( y ) = −τ−1yTx1 + τ−1lse ( τ−1 , X̃Ty ) , where lse is the log-sum-exp function ( see Eq . ( A103 ) in the Appendix ) . The gradient of the InfoNCE loss with respect to y is −τ−1x1 + τ−1Xsoftmax ( τ−1XTy ) and the gradient of the InfoLOOB loss is −τ−1x1 + τ−1X̃softmax ( τ−1X̃Ty ) . Using p = ( p1 , . . . , pN ) T = softmax ( τ−1XTy ) , the gradient of InfoNCE with respect to y is −τ−1 ( 1 − p1 ) ( x1 − X̃softmax ( τ−1X̃Ty ) ) and its gradient with respect to x1 is −τ−1 ( 1− p1 ) y ( see Appendix Subsection A.1.4 ) . By and large , the gradient of InfoNCE is scaled by ( 1− p1 ) compared to the gradient of InfoLOOB , where p1 is softmax similarity between the anchor y and positive sample x1 . Consequently , InfoNCE saturates and learning stalls when anchor and positive sample become similar to each other .

Summary: By using the InfoNCE loss for model training, CLIP has achieved great success. In this paper, the authors propose CLOOB, short for "Contrastive Leave One Out Boost", where modern Hopfield networks are used together with the InfoLOOB objective. InfoLOOB is a leave-one-out upper bound of mutual information, and modern Hopfield networks replace the original embeddings by retrieved embeddings. Results show that CLOOB outperforms CLIP at zero-shot transfer learning across multiple architectures and datasets.