CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
Authors: Andreas Fürst
Elisabeth Rumetshofer
Viet Thuong Tran
Hubert Ramsauer
Fei Tang
Johannes Lehner
D P Kreil
Michael K Kopp
Günter Klambauer
Angela Bitto-Nemling
Sepp Hochreiter
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep learning
Associative memory
Hopfield networks
Contrastive learning
Multimodal learning

Abstract: Contrastive learning with the InfoNCE objective is exceptionally successful in various self-supervised learning tasks. Recently, the CLIP model yielded impressive results on zero-shot transfer learning when using InfoNCE for learning visual representations from natural language supervision. However, InfoNCE as a lower bound on the mutual information has been shown to perform poorly for high mutual information. In contrast, the InfoLOOB upper bound (leave one out bound) works well for high mutual information but suffers from large variance and instabilities. We introduce "Contrastive Leave One Out Boost" (CLOOB), where modern Hopfield networks boost learning with the InfoLOOB objective. Modern Hopfield networks replace the original embeddings by retrieved embeddings in the InfoLOOB objective. The retrieved embeddings give InfoLOOB two assets. Firstly, the retrieved embeddings stabilize InfoLOOB, since they are less noisy and more similar to one another than the original embeddings. Secondly, they are enriched by correlations, since the covariance structure of embeddings is reinforced through retrievals. We compare CLOOB to CLIP after learning on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.

Under review as a conference paper at ICLR 2022
CLOOB: MODERN HOPFIELD NETWORKS WITH INFOLOOB OUTPERFORM CLIP
Anonymous authors Paper under double-blind review
ABSTRACT
Contrastive learning with the InfoNCE objective is exceptionally successful in various self-supervised learning tasks. Recently, the CLIP model yielded im- pressive results on zero-shot transfer learning when using InfoNCE for learning visual representations from natural language supervision. However, InfoNCE as a lower bound on the mutual information has been shown to perform poorly for high mutual information. In contrast, the InfoLOOB upper bound (lcave one out bound) works well for high mutual information but suffers from large variance and instabilities. We introduce "Contrastive Leave One Out Boost" (CLOOB), where modern Hopfield networks boost learning with the InfoLOOB objective. Modern Hopfield networks replace the original embeddings by retrieved embeddings in the InfoLOOB objective. The retrieved embeddings give InfoLOOB two assets. Firstly, the retrieved embeddings stabilize InfoLOOO,, since they are less noisy and more similar to one another than the original embeddings. Secondly, they are enriched by correlations, since the covariance structure of embeddings is reinforced through retrievals. We compare CLOOO CLIP after learning on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.
INTRODUCTION
With the advent of large corpora of unlabeled data in vision and language, self-supervised learning via contrastive learning has become highly successful. Some contrastive learning objectives, such as those of BYO (Grill et al., 2020) and SimSiam (Chen & He, 2021), do not require negative samples. However, the most popular objective for contrastive learning is InfoNCE (van den Oord et al., 2018), in which for anchor sample, positive sample is contrasted with negative samples.
The idea to use objectives with negative samples well known in deep learning (Gutmann & Hyväri- nen, 2010; Chen et al., 2017; Mikolov et al., 2013). For contrastive learning, the most successful objective is InfoNCE, which has been introduced as Contrastive Predictive Coding (CPC) (van den Oord et al., 2018). InfoNCE has been applied to transfer learning (Hénafff al., 2019), to natural language response suggestion (Henderson et al., 2017), to learning sentence representations from unlabelled data (Logeswaran & Lee, 2018), and to unsupervised feature learning by maximizing distinctions between instances (Wu et al., 2018). InfoNCE has been used for learning visual repre- sentations Pretext-Invariant Representation Learning (PIRL) (Misra & vanDerMaaten, 2020), in Momentum Contrast (MoCo) (He et al., 2020), and in SimCLR (Chen et al., 2020). SimCLR became well known as was highly effective for transfer learning. Zero-shot transfer learning (Lampert et al., 2009) is one of the most ambitious goals in vision, since it would improve various real-world downstream applications. Current models natural language processing and vision perform very well standard benchmarks, but they fail at new data, new applications, deployments in the wild, and stress tests (D' Amour et al., 2020; Recht et al., 2019; Taori et al., 2020; Lapuschkin et al., 2019; Geirhos et al., 2020). A model with high zero-shot transfer learning performance will not fail on such data, therefore will be trusted by practitioners.
Pre-training (CLIP) based the InfoNCE objective yielded very impressive results at zero-shot transfer learning (Radford et al., 2021). CLIP learns expressive image embeddings directly from raw text, thereby leverages much richer source of supervision than just
Contrastive Language-Iman

---

Under review as a conference paper at ICLR 2022
labels. A plethora of CLIP follow-up work has already been published (see Appendix Section A.5). The CLIP model is considered as an important foundation model (Bommasani et al., 2021). Though CLIP excels at zero-shot transfer learning, it can be improved.
CLIP training suffers from an "cxplaining away" problem (Wellman & Henrion, 1993), which leads to "shortcut learning" (Geirhos et al., 2020) or the Clever Hans phenomenon (Lapuschkin et al., 2019). Explaining away impedes the increase of the similarity between a text and a corresponding image, since learning focuses on only one common aspect and does not exploit the full covariance structure of the data. If one common aspect is sufficient for high similarity, the InfoNCE objective saturates, since has the form a/(e + b) with a giving the similarity of matched pair and b giving the average similarity of unmatched pairs. For a large similarity a, the objective saturates and increasing a has a small effect. Contrary to InfoNCE, the leave-one-out ("Infol.OOB" bound (Poole et al., 2019) is of the form a/b which does not saturate. However, so far the Infol .OOB bound was not used as an objective in contrastive learning. We justify the maximization of the InfoLOOO bound for contrastive learning in Appendix Section A.1.3. We show that maximizing the InfoLOOB bound leads to good approximation of the mutual information, in particular for high mutual information. A problem of InfoLOOB is that it has high variance for small b.
Even when InfoLOOB avoids saturation, CLIP insufficiently extracts the covariance structure in the data. The covariance originates from co-occurrences of related words in text or from co-occurrences of objects, textures, or colors in images. CLJPP's problem of insufficiently extracting the covariance structure of the data is tackled by modern Hopfield networks. Hopfield networks are energy-based binary associative memories, which popularized artificial neural networks in the 1980s (Hopfield, 1982; 1984). Associative memory networks have been designed store and retrieve samples. Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopfield, 2016). In contrast to these binary memory networks, we use continuous associative memory networks with very high storage capacity. These modern Hopfield networks for deep learning architectures have an energy function with continuous states and can retrieve samples with only one update (Ramsauer et al., 202 2020). Modern Hopfield Networks have already been successfully applied to immune repertoire classification (Widrich et al., 2020) and chemical reaction prediction (Seidl et al., 2021). Modern Hopfield networks reinforce the covariance structure in the data and stabilize the InfoLOOB objective by increasing b.
The covariance structure of retrieved embeddings is amplified through co- co-occurrences of embedding features in the memory. Additionally, the retrieved embeddings are less noisy and more similar to one another which leads to a larger b. We introduce "Contrastive Leave One Out Boost" (CLOOB) which overcomes CLIP's problems of (i) "explaining away" with saturation and (ii) insufficiently extracting the covariance structure of the data. CLOOB uses the leave-one-out ("InfoLOOB") bound (Poole et al., 2019) as the objective in combination with modern Hopfield networks.
Our contributions are:
(a) we introduce new contrastive learning method called CLOOB,
(b) we propose InfoLOOB as an objective for contrastive learning.
(c) we propose to use modern Hopfield networks to reinforce covariance structures,
(d) we show theoretical properties of the InfoLOOB objective and loss function.
2 INFOLOOB vs. INFONCE
We discuss and analyse known bounds on the mutual information 1(X; ; Y) between random variables X and Y, which are distributed according to p(æ;, y):
I(X; ; Y)- = Ep(z.v) In p(x, y) p(x)) p(v)
In p( @
y) p(xr)
In ply x) p(y)
= Ep(e.w)
= Ep(e.))
(1)
We consider the multi-sample lower bound "InfoNCE" (van den Oord al., 2018). A pair of an anchor sample y and a positive sample æ is drawn via the joint distribution p(#1, y). The negative samples X = {æzz.... .NN} are drawn iid according to the marginal distribution p(x). Using x (w), .2. ... NN ). the probabilities of the datasets are x(X) -I;"2 p(xx), p(X X y)
2

---

Under review as a conference paper at ICLR 2022
p(a1 y) II= p(z,), and p II p(x.). The InfoNCE with score function f(x, y) is
[In
f(21, y) N fax
InSONCE((11 Y) = Epry)) Fp(xiw)
N
Ei
using the factor 1/N as in Poole et al. (2019); Tschannen et al. (2019); Cheng et al. (2020); Chen et al. (2021). For far y) ply æ)), we obtain the InfoNCE with probabilities. The InfoNCE is lower bound on the mutual information (Poole et al., 2019), which is stated in the next theorem.
Theorem (InfoNCE lower bound). InfoNCE with score function f(x y) is a lower bound on the mutual information:
Ep(x(w) In f(x1,3) y)
I[ = InNONCE(X Y).
I(X1 Y) 2 Ep(n)
(3)
N Ei= f(2;; y)
In particular, the bound holds for InfoNCE with probabilities, i.e. for f(x, p(y x)).
For a proof see Poole et al. (2019) and the proof of Theorem A1 in the Appendix. The "Leave
upper bound" (Poole et al., 2019) on the mutual information was called "L1Out" in Cheng et al. (2020), while we call it "InfoLOOB" (LOOB for "Leave One Out Bound"). InfoLOOB the same InfoNCE (Eq. (3)), but without the positive sample xi in the denominator. Contrastive Log-ratio Upper Bound (CLUB), another upper bound on the mutual information, was only used for minimizing it (Cheng al., 2020). Maximizing CLUB failed in experiments, because the embedding distribution was not uniform as known for similar objectives (Wang & Liu, 2021). Uniform embedding distributions are required for successful contrastive learning (Wang & Isola, 2020).
We use InfoLOOB as an objective, since it approximates high mutual information better than InfoNCE. Maximizing an upper bound on the mutual information might be counter- intuutr- Therefore, we justify the maximization of the InfoLOOB bound for contrastive learning in Appendix Section A.1.3. We show that maximizing the InfoL.Of bound approximates the mutual information, the better the higher it is. Recently, Infol was independently introduced for and successfully applied to sange-to-imagee contrastive learning (Ych et al., 2021).
The InfoLOOB with score function f(x 3) is defined in the following, where we obtain the In- foLOOB with probabilities for f(x y) = p(y æ):
In
f(x 3) N-T Eez f(x,, y.
Linoo.ooB(ii Y) Ep(r)
Ep(xIT)
(4)
Before we show that InfoLOOB with a score function is an upper bound on the mutual information, we need some definitions. f(x y) draws the positives for y with lower probability than p(æ), that is, the positives are under-sampled. Zyy) Ep(aly) [f y) gives the average score f(x, y). if
under-sampling via p(æ: y). while Z*(y) = Fx((=) [f( v)) average score f(x., y) if sampling
from p(x). We define the variational distribution q( y) = ((x.)) Our main assumption is
(1
expressed by the log-ratio of the averages (3y and Z (y):
Ep(y) [KL(p(æ y) || q(x y)) < Epiz) [In (1) In Z(y)]
(5)
which ensures that the positives are sufficiently under-sampled via p( y). The Hallback-Leiblee divergence gives the minimal difference between averaging f(x y) via p(æ) and via p(æ y). The next theorem shows that InfoLOOB an upper bound on the mutual information.
Theorem (InfoLOOB upper bound). If X {az, ......... ENN are drawn iid according to p(æ y
and if the main assumption Eq. (5) holds, then InfoL.OOB with score function f(x,, y) is an upper bound on the mutual information:
)
f(x y)
I(X1 Y) < Ep(w)
Ep((
In
= LInfoLoO((X, Y). (6)
Ein f(x,, v))
W-I
The bound is valid for InfoLOOB with probabilities (without under inder-sampling, where the negative samples X { x are drawn iid according to p(x:) and f(x = p(y æ:).
The proof for this theorem is given proof for Theorem A2 in the Appendix.
3

---

Under review as a conference paper at ICLR 2022
Loss functions and their gradients. The training set { (()),))), (22, y2). ..... (ZN, yN)] consists of N samples that are drawn iid from p(a, y). InfoNCE uses the matrix X = (x1,.... .N). while InfoLOOB uses X = (#'2.... NN The matrices differ by the positive sample æ1 For the score function f(x y). we use f(z, y) = exp(T siimm-ssmeeieee (x, y)) with the similarity sim y) y x and T as the temperature. We have the InfoNCE and InfoLOOB loss functions:
1
exp(r-1 2 y:)
1
exp(r xz. yi)
LIufNNC =
In
In
(7)
N
exp(r-1 u} yソ)
N
exp 25 ys)
1=
j=1
j=1
LInfoLOOB
1
Σ
In
exp(T-1 x,!y,)
Σ
In
cxp(T x:! y:)
(8)
N
Ljzt exp(x-1 1 x yj)
N 5#
exp(x x y/
In the second sum of the losses in Eq. 7 and Eq. 8, we consider only the first term. For simplicity, we abbreviate y - yı leading the pair (æ y) and the negatives X (IZ2, (, @NN).
exp(T 2 y)
exp(T x y)
LInfONCE(y) = In Σ N jj=1 exp(p= x) y)
LInOLOOB(() = In Ej= exp(T - x y)
These loss terms can be simplified to LinfoNCE()) -r-lyaa Φ1 lse(r= 1xx y) and LinfoLOOB (3)) --- y x1 y- -se((-- ( y). where lse is the log-sum-exp function (see Eq. (A103) in the Appendix). The gradient of the InfoNCE loss with respect to y is ---1an -- Xssftmax((7 x y) and the gradient of the InfoLOOB loss is ---1æ T- Xsoftmaxx( 'x"y). Using p (PI...., PN.. = xoftmax(r-1x" y). the gradient of Pz) - Xsoftmax(7 X y)) and its gradient with
'1 -r-1(1 respect to æ is -1 pıyy (see Appendix Subsection A.1.4).
InfoNCE with respect to y -T
By and large, the gradient of InfoNCE scaled by (1 - p1) compared the gradient of InfOLOOB, where p1 is softmax similarity between the anchor and positive sample æ1. Consequently, InfoNCE saturates and learning stalls when anchor and positive sample become similar to each other.
CLOOB: INFOLOOB WITH MODERN HOPFIELD NETWORKS
image encoder
22
22
2
Hopfield retrieval with U
Hopfield retrieval with V VaN
Un
U=
Un
V=
V
legend
similarity anchor
positive sample
negative sample
Upi Uvg
Upx
Vuy
V
Vay
Hopfield retrieval with U
Hopfield retrieval with
Our dog playing the snow.
y1
y2
yn
y1
y/2
y/
text encoder
Figure 1: The CLOOB architecture for image-text pairs. The image embedding æ; and the text em- bedding y: retrieve the embeddings Uz, and Uy respectively, from a modern Hopfield network that stores image embeddings U (u,,..., uMM) (green boxes at the left). The retrevee image embedding Ua, serves as anchor in order to contrast the positive text-retrieved image embedding Uy, with the negative text-retrieved image embedding Uy, for j f i. Analog, for the second modern Hopfield network that stores text embeddings V (Vi,..., vK) (green boxes at the right).
4

---

Under review as a conference paper at ICLR 2022
CLOOB for contrastive learning. Our novel Contrastive Leave One Out Boost (CLOOB) combines the InfoLOOB objective with modern Hopfield networks. Modern Hopfield networks substitute the original by retrieved embeddings, thereby reduce the variance of InfoLOOB and reinforce the covariance structure in the data. Figure sketches the CLOOB architecture for image-text pairs.
The training set consists of N pairs of embeddings {.11, (eR1,y)), y1), (æN, (ENy yN)). M stored embeddings U = (u),.... M. ). and K stored embeddings V = (V)..... Ug). ). The state or query embeddings 2; and y, retrieve Un: and Ug, respectively, from U analogous notation for retrievals from V.
All samples are normalized: |x::|| = | = || ||0:|| 1. The following vectors are retrieved from modern Hopfield networks (Ramsauer et al., 2021):
U2: U softmax(3 U UTx:). xx),
Uy U softmax(3 U y:)
(9)
V2: = softmax( V xx;;).
Vu V softmax(3 V (Ty:)
(10)
where Un, denotes an image-retrieved image embedding, Uy, text-retrieved image embedding, Va, an image-retrieved text embedding and Vy, text-retrieved text embedding. The hyperparameter corresponds the inverse temperature: B retrieves the average of the stored pattern, while large / retrieves the stored pattern that is most similar to the state pattern (query).
In InfoLOOB, CLOOB substitutes the embedded samples æ; and y: by the retrieved embedded samples. In the first term, Z; and y: are substituted by Ua, and U respectively. while in the second term by Vz, and Vy All retrieved samples are normalized, [[ 1| V. = VV, We obtain the InfoLOOO loss function that is used by CLOOB:
LLnfLLOB
1 N
Σ 11
In
exp(T UT UN)
m.
1
Σ
N
In exp(T VT
Ei exp(T UT
N
t= j exp(T VT V
(11)
Modern Hopfield Networks reduce high variance of InfoLOOB.. CLOOB uses InfoL.OO as objective, since it estimates the mutual information (MI) better than InfoNCE, in partieular, for large MI. Cheng et al. (2020, Fig. 1 and Fig. 2) show that InfoLOOB is better estimator for the MI than InfoNCE (van den Oord al., 2018), MINE (Belghazi al., 2018), and NWJ (Nguyen et al., 2010). We experimentally confirmed that InfoLOOB better estimates the mutual information than InfoNCE.
without Hopfield
with Hopfield
without Hopfield
with Hopfield
MI
Vaniing Sereee
Tr Ml
X1
1214
Figure 2: Variance reduction of InfoLOOB by modern Hopfield networks. From left to right: without Hopfield for MI 10, with Hopfield for MI 10, without Hopfield for MI 14, with Hopfield for MI 14. Modern Hopfield networks reduce the variance of the InfoLOOB loss.
However, InfoLOOB has higher variance than lower bounds on MI like InfoNCE, which considerably hampers learning (Cheng al., 2020, Fig. 1 and Fig. 2). see also Appendix Section A.2. The InfoNCE objective has the form a/(a + b) while InfoLOOB has the form a/b with a giving the anchor-to-positiv similarity and the average anchor-to-negative similarity. For small b, we observe high variance and instability of InfoLOOO. Modern Hopfield networks (Ramsauer et al., 2021) are a remedy for the high variance. Modern Hopfield networks substitute the original patterns by retrieved patterns, which are an average over the stored patterns. We tested the variance of MI estimators/bounds on toy tasks, with samples drawn from Gaussian distributions following (Belghazi et al., 2018; Poole al., 2019; Cheng et al., 2020). With the InfoLOOB objective, train deep learning architectures with and without modern Hopfield networks on top, where the current learning batch is stored in the modern Hopfield networks. We used training data with mutual information of
5

---

Under review as a conference paper at ICLR 2022
10 and 14, where the parameters were optimized for the best performance a validation set. We test the final model on different levels of mutual information. Figure 2 shows that modern Hopfield networks reduce the variance of the model. The average variances are reduced from 0.67 to 0.33 for MI 10 and from 1.00 to 0.48 for MI 14 (more details in Appendix A.2).
Modern Hopfield Networks amplify the covariance structure in the data. The covariance struc- ture is extracted by the retrieved embeddings Uz Uy, and V2,Ya Vye- The Jacobian J of the soft- max p = (5f) is ((Ba) = B (diag(p) - -pp"). We define the weighted covariance COv(U), where sample u; is drawn with probability ps. [Cov(U)]ą = (UJ(Ba)U')) IN 3 1 44 - - i Σ p.ua). The formula of the weighted covariance differs from the standard empirical covariance, since the factor 1/M replaced by p:- Thus u, is sampled with probability p: instead of being sampled uniformly with probability 1/M.
We apply the mean value theorem to the softmax function with mean Jacobian matrix J J''(aa)
Io ((Ba) dA. The mean Jacobian Jm (3a) is a symmetric, diagonally dominant, positive semi- definite matrix with one eigenvalue of zero for eigenvector 1 and spectral norm bounded by ||''|la < 0.5,3 (see Appendix Lemma A1). We can express m. Uy. as (see Appendix Theorem A3):
UH Uy.
(ū + Cov(U, x;)" (u + Cov(U, .y.) y.)
(12)
where the mean is ū 1/MU1 and the weighted covariances are Cov(U, ) UJ" ,,) and Cov(U, ya) = UJU""""""" ys U.) The weighted covariance Cov(U, is the covariance if the stored pattern u, is drawn according an averaged p: given by J')... When maximizing the dot product U Uy. the normalized vectors 2; and y; are encouraged to agree on drawing the patterns u, with the same probability p; in order to generate similar weighted covariance matrices Cov (U,)). If subsets of U have a strong covariance structure, then it can be exploited to produce large weighted covariances and, in turn, large dot products of U Ug. Furthermore, for large dot product U² a, U x; and y: have to be similar to each other to extract the same direction from the covariance matrices. Above considerations for UT! a; Uy analogously apply to V m; Vg..
æ. Vy., because these dot products We did not use loss function that contains dot products like UT have higher variance than the ones we have used. The dot product UT, Vy, has higher variance, since it uses M + K stored patterns, whereas UT Uy, and V Vy: use M and K, respectively.
Modern Hopfield Networks can reuse training samples as stored patterns. We use the training samples as the stored patterns in the modern Hopfield network. Hence, we set u, æ; and v, = yi. that is, U X and V Y. Consequently, we store the learning batch in the modern Hopfield networks as U and V. In particular this means that z; can retrieve itself from U = X but not from
V V=X Y. Analogously, y: can retrieve itself from V Y but not from U X.
Modern Hopfield networks allow the usage of retrieved embeddings, After learning, both the model embeddings x and y as well as the retrieved embeddings U.. Uy. Va, and Vy may serve for the downstream tasks, e.g. for zero-shot transfer learning. When using the retrieved embeddings, the modern Hopfield networks can store random samples, prototypes, templates, or proprietary samples. Therefore, particular embedding features can be amplified according to the task at hand.
Modern Hopfield networks a new concept for contrastive learning. bioinformatics the covariance structure in a sequence is reinforced by first retrieving similar sequenees from database and then aligning them. Conserved regions are characterized by high local covariance in the alignment (Dickson & Gloor, 2012; Kreth & Fodor, 2014). Modern Hopfield networks detect high covariances of embedded features, which is conveyed by the retrieved sample that corresponds to an alignment.
4 EXPERIMENTS
On two pretraining datasets, we compare our new CLOOB to CLIP (Radford et al., 2021) with respect to their capability of zero-shot transfer learning. The first dataset, Conceptual Captions (CC) (Sharma et al., 2018), has a very rich textual description of images but only three million image-text pairs. The second dataset, subset of YFCC100M (Thomee et al., 2016), has 15 million image-text pairs but the textual description is less rich than for CC and often vacuous. For both pretraining datasets, the downstream zero-shot transfer learning performance is tested seven image classification datasets.

---

Under review as a conference paper at ICLR 2022
Table 1: Zero-shot results for models trained on CC with ResNet-50 vision encoders for two different checkpoints. Results are given mean accuracy over 5 runs. Statistically significant results are shown in bold. CLIP and CLOOB were trained for 31 epochs while CLIP* and CLOOB* were trained for 128 epochs. In the majority of tasks CLOOB significantly outperforms CLIP.
CLIP RN-50
CLOOB RN-50
CLIP*
CLOOB RN-50
Dataset
RN-50
Birdsnap
2.26 = 0.20
3,06 = 0.30
2.8 + 0.16
3,24 0.31
Country211
0.67 + 0.11
0.67 + 0.05
4.7::.04
0.73 + 0.05
Flowers 102
12.56 + 0.38
13.45 1.19
13.32
14.36 + 1.17
GTSRB
7.66 1.07
6.38
8.96
7.03 1.22
UCF101
20.98 1.55
0.72
21.63
23.03 0.85
Stanford Cars
0,91 + 0.10
1.23
0.16
1.41 0.32
ImageNet
20.33 + 0.28
23.97
0..2
25.67 + 0.22
ImageNet V2
20.24 + 0.50
23.59 0.15
21.24 0.22
25.49 0.11
4.1 CONCEPTUAL CAPTIONS PRETRAINING
Pretraining dataset. The Conceptual Captions (CC) (Sharma et al., 2018) dataset consists of 2.9 million images with high-quality captions. Images and their captions have been gathered via an automated process from the web and therefore represent wide variety of content. Raw descriptions of images are collected from the alt-text HTML attribute. Both images and texts are filtered for high quality image-text pairs.
We compare our new CLOOB to CLIP (Radford et al., 2021). The CLOOB implementation is based on OpenCLIP (Ilharco et al., 2021), which achieves results equivalent to CLIP on the YFCC dataset (see Section 4.2). OpenCLIP also reports results on the CC dataset. As CLIP does not train models on CC we report results from this reimplementation as baseline. Analogously to Radford et al. (2021, Section 2.4), we use the modified ResNet (He et al., 2016) and BERT (Devlin et al., 2018; 2019) architectures encode image and text input. We use the ResNet encoders ResNet-50, ResNet- 101, and ResNet-50x4.
Methods compared.
Hyperparameter selection and learning schedule. We use the hyperparameter values of OpenCLIP, concretely, a learming rate of 1 x 10-3 and weight decay of 0.1 for the Adam optimizer (Kingma al., 2014) with decoupled weight decay regularization (Loshchilov Hutter, 2019). Deviating from OpenCLIP, we use batch size of 512 due to computational restraints, which did not change the performance. The learning rate scheduler for all experiments is cosine annealing with warmup and hard restarts (Loshchilov & Hutter, 2017). We report the hyperparameter 7 (default 0.07) from CLIP as 7- of 14.3 to be the same regime as the hyperparameter 3 for the modern Hopfield networks. The main hyperparameter search for CLOOB (also for YFCC pretraining in the next section) was done with ResNet-50 as the vision encoder. Learnable T -1 in combination with the InfoLOOB loss results in undesired learning behavior (see Appendix Section A. Therefore, we set - to fixed value of 30, which was determined via hyperparameter search (see Appendix Section A.3.2). For modern Hopfield networks,the hyperparameter 33 was set Further we scale the loss in Eq. (11) with 7 to remove the factor -1 from the gradients (see Appendix Section A.1.4) resulting in the loss function TLLLoLLOB--
Evaluation metrics: Zero-shot transfer learning. We evaluate and compare both CLIP and CLOOB on their zero-shot transfer learning capabilities on the following downstream image classi- fication tasks. Birdsnap (Berg et al., 2014) contains images of 500 different North American bird species. The Country211 1 (Radford al., 2021) dataset consists of photos across 211 countries and is designed to test the geolocalization capability of visual representations. Flowers (Nilsback & Zisserman, 2008) is a dataset containing images of 102 flower species. GTSRB (Stallkamp et al., 2011) 1) contains images for classification of German traffic signs. UCFI01 (Soomro et al., 2012) is video dataset with short clips for action recognition. For UCF101 we follow the procedure reported in CLIP and extract the middle frame of every video assemble the dataset. Stanford Cars (Krause et al., 2013) contains images of 196 types of cars. ImageNet (Deng et al., 2009) is a large scale image classification dataset with images across 1,000 classes. ImageNetv2 (Recht et al., 2019) consists of
7

---

Under review as a conference paper at ICLR 2022
Table 2: Performance with InfoLOOB vs. InfoNCE objective and with vs, without Hopfield retrieval. InfoLOOB increases the performance of CLIP in most of the tasks. Hopfield with InfoLOOB strongly improves the performance in 7 out of 8 datasets compared both CLIP models.
CLIP
Hopfield
Dataset
InfoNCE
InfoLOOB
InfoNCE
InfoLOOB
Birdsnap
1.94
2.37
1.67
2.53
Country211
0.62
0.63
0.54
0.76
Flowers104
13.04
13.03
11.53
14.24
GTSRB
7.28
4.39
5.76
5.86
UCF101
21.00
19.14
20.56
22.29
Stanford Cars
0.90
1.33
1.24
1.37
ImageNet
20.31
22.13
19.04
24.21
ImageNetV2
20.63
21.65
18.97
23.80
three new test sets with 10,000 images each for the ImageNet benchmark. For further details see Appendix Section A.3.3.
Results. We employ the same evaluation strategy and use the prompt templates as published CLIP (see Appendix Section A.3.3). We report zero-shot results from two checkpoints in Table 1. CLIP and CLOOB were trained for comparable number of epochs used in CL.PP (see Appendix Section A.3.2) while CLIP* and CL.OOB* were trained until evaluation performance plateaued (epoch 128). In both cases CLOOB significantly outperforms CLIP on the majority of tasks or matches its performance. Statistical significance of these results was assessed by an unpaired Wilcoxon test on a 5% level.
Ablation studies. CLOOB has two new major components compared to CLIP: (1) the InfoLOOB objective instead of the InfoNCE objective and (2) the modern Hopfield networks. To assess which of the new major components of CLOOB has led to the performance increase over CLIP, we performed ablation studies on CC. First, we enhanced CL.IP by replacing the InfoNCE objective with InfoL.OBB Table 2 shows that the InfoLOOOB objective increases the performance of CLIP in the majority of the datasets. The reason is that InfoLOOB suffers less than InfoNCE from the "explaining away" problem. However, InfoLOOB is more effective for higher mutual information, that is, for richer covariance structure. Hopfield networks amplify the covariance structure by retrieved embeddings. For InfoLOOB, however, this amplification is disadvantageous the saturation effect is increased by higher similarity between anchor and positive. Thus, combining modern Hopfield networks with InfoNCE leads to performance drop. Combining Hopfield and InfoLOOB into CLOOB strongly improves the performance on 7 out of zero-shot transfer learning tasks. An additional ablation considers the learning rate scheduler. For more details see in Appendix Section A.3.1.
4.2 YFCC PRETRAINING
Pretraining dataset. To be comparable the CLIP results, samples from the TFCC100M dataset (Thomee et al., 2016) as in Radford et al. (2021), which we refer to YFCC. YFCC was created by filtering YFCC100M for images which contain natural language descriptions and/or titles in English. It was not filtered by quality of the captions, therefore the textual descriptions are less rich and contain superfluous information. The dataset with 400 million samples used to train the CLIP models in Radford et al. (2021) has not been released and, thus, not available for comparison. Due to limited computational resources we are unable to compare CLOOB to CLIP on other datasets of this size.
the same subset of 15 million
Methods compared and evaluation. In addition to the comparison of CLOOB and CLIP based on the OpenCLIP reimplementation (Ilharco et al., 2021), we include the original CLIP results (Radford et al., 2021, Table 12).
Hyperparameter selection. We use the hyperparameters selected at the Conceptual Captions dataset, except learning rate, batch size, and B. For modern Hopfield networks, the hyperparameter 3 is set to 14.3, which is the default parameter of 7 for the InfoNCE objective in Radford et al. (2021). Furthermore, the learning rate set to 5 10 and a batch size 1024 as in OpenCLIP of Ilharco al. (2021). ). For further details see Appendix Section A.3.2.
8

---

Under review as a conference paper at ICLR 2022
Evaluation metries. As in the previous experiment, methods again evaluated their zero-shot transfer learning capabilities on downstream tasks.
Results. Table 3 provides results of the original CLIP and CLOOB trained on YFCC. The results on zero-shot downstream tasks show that CLOOB outperforms the results CLIP on all 7 tasks (ImageNet V2 results have not been reported in Radford et al. (2021)). Similarly, CLOOB outperforms CLIP on out of tasks for linear probing. Results of the comparison of CLOOB an the CLIP reimplementation of OpenCLIP are given in Table 4. CLOOB exceeds the CLIP reimplementation in 7 out of 8 tasks for zero-shot classification using ResNet-50 encoders. With larger ResNet encoders, CLOOB outperforms CLIP on all tasks. Furthermore, the experiments with larger vision encoder networks show that CLOOB performance increases with network size. Visualizations of predictions of CLOOBB zero-shot classifiers from all datasets are shown in Appendix Section A.3.4.
Table Results of CLIP and CLOOB trained on YFCC with ResNet-50 encoder. Except for one linear probing dataset, CLOOB consistently outperforms CLIP across all tasks.
Linear Probing CLIP
Zero-Shot
CLÖOB
CLIP (OpenAl)
CLOOB
Dataset
(OpenAI)
(ours)
(ours)
Birdsnap
47.4
56.2
19.9
28.9
Country211
23.1
20.6
5.2
7.9
Flowers102
94.4
96.1
48.6
55.1
GTSRB
66.8
78.9
6.9
8.1
UCF101
69.2
72.3
22.9
25.3
Stanford Cars
31.4
37.7
3.8
4.1
ImageNet
62.0
65.7
31.3
35.7
ImageNet V2
58.7
34.6
Table 4: Zero-shot results for the CLIP reimplementation and CLOOB using different ResNet architectures trained on YFCC. CLOOB outperforms CLIP in 7 out of tasks using ResNet-50 encoders. With larger ResNet encoders CLOOB outperforms CLIP all tasks. The performance of CLOOB scales with increased encoder size.
CLIP RN-50
CLOOB RN-50
CLIP
CLOOB RN-101
CLIP
CLOOB
Dataset
RN-101
RN-50x4
RN-50x4
Birdsnap
21.8
28.9
22.6
30.3
20.8
32.0
Country211
6.9
7.9
7.8
8.5
8.1
9.3
Flowers102
48.0
55.1
48.0
55.3
50.1
54.3
GTSRB
7.9
8.1
7.4
11.6
9,4
11.8
UCF101
27.2
25.3
28.6
28.8
31.0
31.9
Stanford Cars
3.7
4.1
3.8
5.5
3.5
6.1
ImageNet
34.6
35.7
35.3
37.1
37.7
39.0
ImageNet V2
33.4
34.6
34.1
35.6
35.9
37.3
CONCLUSION
For constrastive learning, have introduced "Contrastive Leave One Out Boost" (CLOOB), for which modern Hopfield networks boost learning with the InfoLOOB objective. Modern Hopfield networks both increase the stability of InfoL.OO and reinforce the covariance structure of the data. We have shown theoretical properties of the InfoLOOB bound and objective. Our results suggest InfoLOOB as alternative InfoNCE in contrastive learning. An ablation study shows that both, the InfoLOOB objective and modern Hopfield networks, are necessary to yield high performance. At seven zero-shot transfer learning tasks, the novel CLOOB is compared CLIP after pretraining on Conceptual Captions and the YFCC dataset. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We will publish the source code after the reviewing period. This will ensure that the results are reproducible in their entirety. The datasets used for training our models well as for the downstream tasks are publicly available.
ETHICAL CONSIDERATIONS
Impact on ML and related scientific fields. Our research has the potential to positively impact a wide variety of fields of life due its general applicability. Most importantly, it has the potential to reduce the cost for training other Al systems, which could lead to reduction compute costs and carbon dioxide emissions.
However, any new development in machine learning can be applied for good or for bad. Our system be used for medical applications where it could save lives but might also be used for surveillance and malevolent systems.
Impact on society. potential danger could arise from an application of our approach in which users rely overly on the outcomes. For example, in a medical setting, physicians might rely on the technical system and shift the liability towards the machine. This might also happen in the domain of self-driving cars, when drivers start paying less attention to the traffic because of an Al-based driving system. Finally, our method may also be deployed companies to automate various simple tasks, which might lead to reduced need for particular jobs in production systems.
Consequences of failures the method. Depending on the application area, a failure of this method might be of lesser concern, such as failed execution of computer program. If our method is employed within a larger automation system, a failure could result in damages such as a car accident or errors of production system. However, this holds for almost all machine learning methods, and their usage and testing depends on the application area.
Leveraging of biases in the data and potential discrimination. Our proposed method relies on human-annotated data and thereby human decisions, which are usually strongly biased. The undesirable biases contained in dataset are learned and may propagate downstream applications. Therefore, the responsible use of our method depends on careful selection of the training data and awareness of the potential biases within those.
REFERENCES
L. F. Abbott and Y. Arian. Storage capacity of generalized networks. Phys. Rev. A, 36:5091-5094 1987. doi: 10. 10.1103PhyyReeA..66.0011
S. Agarwal, G. Krueger, J. Clark, A. Radford, J. W. Kim, and M. Brundage. Evaluating CLIP: towards characterization broader capabilities and downstream implications. ArXiv, 2108.028 02818, 2021.
Baldi and S. S. Venkatesh. Number of stable points for spin-glasses and neural networks of higher orders. Phys. Rev. Lett., 58:913-916 1987. doi: 10. 10.1103/hhyRReveeett.....9
D. Bau, A. Andonian, A. Cui, Y Park, A. Jahanian, A. Oliva, and A. Torralba. Paint by word. arXiv preprint arXiv:2103. 10951, 2021.
M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Bengio, A. Courville, and R. D. Hjelm. Mutual information neural estimation. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pp. 531-540 PMLR, 2018.
T. Berg, Liu, S. W. Lee, M. Alexander, D. W. Jacobs, and N. Belhumeur. Birdsnap: Large-scale fine- grained visual categorization of birds. In Proc. Conf. Computer Vision and Pattern Recognition (CVPR), pp. 2019-2026, 2014. doi: 10. 2.1109/VVPR.204.220
R. Bommasani al. On the opportunities and risks of foundation models. ArXiv, 2108.07258, 2021.
10

---

Under review as a conference paper at ICLR 2022
Q. Cai, Y. Wang, Y. Pan, T. Yao, and T. Mei. Joint contrastive learning with infinite possibilities. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 28638-12648 Curran Associates, Inc., 2020.
B. Caputo and H. Niemann. Storage capacity of kemel associative memories. In Proceedings of the International Conference Artificial Neural Nenworks (ICANN), pp. 51-56, Berlin, Heidelberg, 2002. Springer-Verley Verlag.
N. Carlini and A. Terzis. Poisoning and backdooring contrastive learning, ArXiv, 2106. 2106.09667, 2021.
H. H. Chen, Y. C. Lee, G. Z. Sun, Lee, T. Maxwell, and Lee Giles. High order correlation model for associative memory. AIP Conference Proceedings, 151(1):86 99, 1986. doi: 10.1063/1. 36224.
J. Chen, Z. Gan, X. Li, Q. Guo. L. Chen, S. Gao, T. Chung, Y. Xu, B. Zeng, W. Lu, F. Li, L. Carin, and C. Tao. Simpler, faster, stronger: Breaking the log-K curse on contrastive learners with FlatNCE. arXiv, 2107.01 0.07001155 152, 2021.
Chen, Sun, Shi, and L. Hong. On sampling strategies for neural network-based collaborative filtering. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 767-776, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.1 141//3099993333992222
T. Chen, Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In H. Daumé and A. Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020.
X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15750-15758. 2021.
P. Cheng, W. Hao, S. Dai, J. Liu, Z. Gan, and L. Carin. CLUB: A contrastive log-ratio upper bound of mutual information. In H. Daume and A. Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 1779-1788. PMLR, 2020.
A. D' A'mour al. Underspecification presents challenges for credibility in modern machine learning. ArXiv, 2011 1. 033955, 2020.
J. Deng, W. Dong, R. Socher, L.-J K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 2248255. leee, 2009.
B. Devillers, R. Bielawski, B. Choski, and R. VanRullen. Does language help generalization vision models? ArXiv, 2104.08313, 2021.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding, ArXiv, 2018.
Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional trans- formers for language understanding. In Proceedings the 2019 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019. doi: 10. 10.18653/11/N19-1422
R. J. Dickson and G. B. Gloor. Protein sequence alignment analysis by local covariation: coevolution statistics detect benchmark alignment errors. PLoS One, 7(6):237645, 2012. doi: 10.1371/journa 0037640.
H. Fang, P. Xiong, L. Xu, and Y. Chen. CLIP2Video: mastering video-text retrieval via image CLIP. ArXiv, 2106.1 1097, 2021.
11

---

Under review as a conference paper at ICLR 2022
K. Frans, L. B. Soros, and O. Witkowski. CLIPDraw: exploring text-to-drawing synthesis through language-image encoders. ArXiv, 2106. 14843, 2021.
A. Galatolo, M. G. C. A. Cimino, and G. Vaglini. Generating images from caption and vice versa via CLIP-guided generative latent space search. ArXiv, 2102.01645, 2021.
B. Gao and L. Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. ArXiv, 2017.
T. Gao, X. Yao, and D. Chen. SimCSE: simple contrastive learning of sentence embeddings. ArXiv, 2104.08821, 2021.
E. Gardner. Multiconnected neural network models. Journal of Physics A, 20(11):3453-3444 1987. doi: 10. 00.108800305444002200 1/046.
R. Geirhos, J. Jacobsen, C. Michaelis, R. S. Zemel, W. Brendel, M. Bethge, and F, A. Wichmann. Shortcut learning in deep neural networks. ArXiv, 2004.07780, 2020.
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Ávila Pires, D. Guo, M. Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your approach sulf-super learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33,
own latent
pp. 21271-21284, Curran Associates, Inc., 2020.
M. Gutmann and Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnor- malized statistical models. In Y. W. Teh and M. Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
T. Han, W. Xie, and A. Zisserman. Self-supervised co-training for video representation learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5679-5690. Curran Associates, Inc., 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
O. J. Hénaff, A. Srinivas. J. DeFauw, A. Razavi, C. Doersch, S. M. A. Eslami, and A. vanDenOord. Data-efficient image recognition with contrastive predictive coding. ArXiv, 1905.09272, 2019.
M. L. Henderson, R. Al-Rfou, B. Strope, Y.-H. Sung, L. Lukáes, R. Guo, Kumar, B. Miklos, and R. Kurzweil. Efficient natural language response suggestion for smart reply. ArXiv, 1705.00652, 2017.
J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554-2555 2558, 1982.
J. J. Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, $180033088-3002 1984. doi: 0t733/ma..81.1..3000
D. Horn and M. Usher. Capacities of multiconnected memory models. J. Phys. France, 49(3): 389-395, 1988. doi: 10. 10..///jhys011188849000000
G. Ilharco, M. Wortsman, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L.. Schmidt. OpenCLIP, 2021.
D. P. Kingma, S. Mohamed, D J. Rezende, and M. Welling. Semi-supervised learning with deep gen- erative models. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances Neural Information Processing Systems 27, pp. 3581-3589, Curran Associates, Inc., 2014.
12

---

Under review as a conference paper at ICLR 2022
J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3D object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), 2013.
K. E. Kreth and A. A. Fodor. Covariance in protein multiple sequence alignments using groups of columns. ArXiv, 1401.1141, 2014.
D. Krotov and J. Hopfield. Dense associative memory for pattern recognition. In D. D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, pp. 1172-1180. Curran Associates, Inc., 2016.
C. H. Lampert, H. Nickisch, and S. Harmeling. Learning to detect unseen object classes by between- class attribute transfer. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 951-958. IEEE, 2009.
Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller. Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications, 10, 2019. doi: 1..10388/41467--111 467-019-08987-4
J. Li, Zhou, C. Xiong, R. Socher, and C. Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations (ICLR), 2021. URL net/ seeeummidd-nny kpuSr jeq. ArXiv 2005.04966.
L. Logeswaran and H. Lee. An efficient framework for learning sentence representations. In Sixth International Conference Learning Representations (ICLR), 2018. URL https:// openreview net forum?idddioo wwwemmdddddddddddo ArXiv 1803.02893.
I. Loshchiloy and F. Hutter. SGDR: stochastic gradient descent with warm restarts. 5th International Conference on Learning Representations ICLR. OpenReview.net, 2017. URL
.nt wttpe//wwww.eettouuuiii--------e meet:/foommm..d.dgg55Ssss
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. URL httpp:://ppeneeeeeeeeeee Bkg6RiCqY7.
net. forum?id
H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. L.. CLIP4Clip: an empirical study of CLIPP for end to end video clip retrieval. ArXiv, 2104.08860, 2021.
D. McAllester and Stratos. Formal limitations on the measurement of mutual information. ArXiv, 1811 1.00251, 2018.
D. McAllester and K. Stratos. Formal limitations on the measurement of mutual information. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings Machine Learning Research, pp. 875-884. PMLR, 26-28 Aug 2020. URL https: http:///ooocccceccooooooooocc ngs. .mlr. malr.a w.t.pres/vvlll v188// mcal lester20a 0a html.
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionaliit. In J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances Neural Information Processing Systems, volume 26, pp. 3111-3119, 19, Curran Associates, Inc., 2013.
T. Milbich, K. Roth, S. Sinha, L. Schmidt, M. Ghassemi, and B. Ommer. Characterizing generalization under
shifts in deep metric learning. ArXiv, 2107 09562, 2021.
Miller, Taori, A. Raghunathan, S. Sagawa, W. Koh, Shankar, Liang, Y. Carmon, and L. Schmidt. Accuracy the line: On the strong correlation between dut-of-distributiin and in-distribution generalization. ArXiv, 2107.04649 2021.
I. Mísra and L. vanDerMaaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
M. Narasimhan, A. Rohrbach, and T. Darrell, CLIP-It! Linguage-guided video summarization. ArXiv, 2107 1007000550, 2021.
13

---

Under review as a conference paper at ICLR 2022
X. Nguyen, M. J. Wainwright, and M. Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. IEEE Transactions on Information Theory, 56(11): 5847-5861, 2010. doi: 10.1 10.1109//2.20000226887770
M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the 2008 Sixth Indian Conference Computer Vision, Graphics and Image Processing, pp- 722-729. IEEE Computer Society, 2008. doi: 10. 1..1199CGGIIP.200.44444
F. W. J. Olver, D. W. Lozier, R. F. Boisvert, and C. W. Clark. NIST handbook of mathematical functions. Cambridge University Press, 1 pap/edr edition, 2010. ISBN 9780521 192255.
D. Pakhomov, S. Hira, N. Wagle. K. E. Green, and N. Navab. Segmentation in style: Unsupervised semantic image segmentation with stylegan and CLIP. ArXiv, 2107. 2021.
B. Poole, S. Ozair, A. vanDenOord, A. A. Alemi, and G. Tucker. On variational bounds of mutual information. In K. Chaudhuri and R. Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5171-5180. PMLR, 2019.
D. Psaltis and H. P. Cheol. Nonlinear discriminant functions and associative memories. AIP Conference Proceedings, 151(1):370-375 1986. doi: 10. 10.1041/1.32241.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference Machine Learning (ICML), 2021.
H. Ramsauer, B. Schäfl, J. Lehner, P. Scidl, M. Widrich, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, V. Greiff, D. Kreil, M. Kopp. G. Klambauer, Brandstetter, and S. Hochreiter. Hopfield networks is all you need. ArXiv, 2008 002217, 2020.
H. Ramsauer, B. Schäfl, J. Lehner, Seidl, M. Widrich, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, V. Greiff, D. Kreil, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. Hopfield networks all you need. In 9th International Conference on Learning Representations (ICLR), 2021. URL https: wwwpreveew....ep/.//pp net/
B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do ImageNet classifiers generalize to ImageNet? In K. Chaudhuri and R. Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5389-5400. PMLR, 2019.
Seidl, P. Renz, N. Dyubankova, Neves, J. Verhoeven, J. K. Wegner, S. Hochreiter, and G. Klam- bauer. Modern hopfield networks for few- and zero-shot reaction prediction. ArXiv, 2104.03279, 2021.
P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.-W. Chang, Z. Yao, and K. Keutzer. How much can CLIP benefit vision-and-languagge tasks? ArXiv, 2107.06383 2021.
K. Soomro, A. R. Zamir, and M. Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11), 2012.
J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The German traffic sign recognition benchmark: A multi-class classification competition. The 2011 International Joint Conference on Neural Networks, pp. 1453 1460, 2011.
R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18583-18599. Curran Associates, Inc., 2020.
14

---

Under review as a conference paper at ICLR 2022
B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. YFCC100M: The new data in multimedia research. Commun, ACM, 59(2):64-73, 2016. doi: 10.1145/2812802.
Y.HH. H. Tsai, M. Q. Ma, H. Zhao, K. Zhang, L.-P. Morency, and R. Salakhutdinov. Conditional contrastive learning: Removing undesirable information in self-supervised representations. ArXiv, 2106.02866, 2021.
M. Tschannen, J. Djolonga, tion maximization for representation learning. arXiv, 1907.13623 2019. URL https: / //openreviie ew net wtte:/ooommddddkoooo2e22 Representations (ICLR).
Rubenstein, S. Gelly, and M. Lucic. On mutual informa-
8th International Conference on Learning
A. den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. ArXiv, 0074003748 2018.
M.J. Wainwright. Basic tail and concentration bounds, pp. 21-57. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/9781108627777
Wang and H. Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/VVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2495-2504, 2021.
T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning (ICMI., 2020.
M. P. Wellman and M. Henrion. Explaining 'explaining away'. IEEE Trans. Pattern Anal. Mach. Intell., 15(3):287-292 1993. doi: 10.1109/34.204111
M. Widrich, B. Schäfl, M. Pavlović, H. Ramsauer, L. Gruber, M. Holzleitner, J. Brandstetter, G. K. Sandve, V. Greiff, S. Hochreiter, and G. Klambauer. Modern Hopfield networks and attention for immune repertoire classification. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18832-18845 Curran Associates, Inc., 2020.
M. Wortsman, G. Ilharco, M. Li, J. W. Kim, H. Hajishirzi, A. Farhadi, H. Namkoong, and Schmidt. Robust fine-tuning of zero-shor shot models. ArXiv, 2109.01903, 2021.
M. Wu, M. Mosse, C. Zhuang, D. Yamins, and N. Goodman. Conditional negative sampling for contrastive learning of visual representations. In International Conference on Learning Represen- tations (ICLR), 2021. URL https: /wwp//eeep.eeeeeee////e/e. net forum?id-v8b3e 6
Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non- paramerrei instance discrimination. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3733-3742, Los Alamitos, CA, USA, 2018. IEEE Computer Society. doi: 10.1 109/ CVPR.2018.0033
C.-H. Yeh, C.-Y. Hong, Y.-C. Hsu, T.-L, Liu, Y. Chen, and Y. LeCun. Decoupled contrastive learning. ArXiv, 21 2110.06848, 2021.
K. Zhou, J. Yang. C. C. Loy, and Z. Liu. Learning to prompt for lingunelanguage models. ArXiv, 2109.01134, 2021.
15

---

Under review as a conference paper at ICLR 2022
A APPENDIX
This appendix consists of four sections (A.1-A.4). Section A. provides the theoretical properties of the InfoLOOB and InfoNCE. It shown how derive that InfoNCE is a lower bound on mutual information. Further is shown how to derive that InfoLOOB an upper bound on mutual information. The proposed loss function LintLLOOB and its gradients discussed. In Section A.2 we discuss the estimation of mutual information for toy example. Section A.3 provides details on the experiments for Section Section A.4 briefly reviews continuous modern Hopfield networks. Section A.5 discusses further related work.
CONTENTS OF THE APPENDIX
A Appendix
16
A.1 InfoLOOB vs. InfoNCE
17
A.1.1 InfoNCE: Lower Bound on Mutual Information
17
A.1.2 InfoLOOB: Upper Bound on Mutual Information
21
A.1.3 InfoLOOB: Analysis of the Objective
25
A.1.4 InfoNCE and InfoLOOB: Gradients
32
A.1.5 InfoLOOB and InfoNCE: Probability Estimators
................................
34
A.1.6 InfoLOOB and InfoNCE: Losses
36
A.2 Mutual Information Estimation
................................
39
A.3
Experiments..
39
A.3.1 Ablation studies
A.3.2 Hyperparameters
41
A.3.3 Datasets
41
A.3,4 Zero-shot evaluation
42
A.3.5 Linear probing
42
A.4 Review of Modern Hopfield Networks
43
A.5 Further Related Work
46
LIST OF THEOREMS
AI
Theorem (InfoNCE lower bound).
19
A2
Theorem (InfoLOOP upper bound)
23
A3
Theorem (Weighted Covariances)
A4
Theorem (Modern Hopfield Networks: Retrieval with One Update)
45
A5
Theorem (Modern Hopfield Networks: Exponential Storage Capacity)
LIST OF DEFINITIONS
A1
Definition (Pattern Stored and Retrieved).
45
LIST OF FIGURES
A1 Estimated mutual information of different objectives
40
A2 Visualization of zero-shot classification of three examples from each dataset
LIST OF TABLES
Al Influence of loss functions and Hopfield retrieval
40
A2 Influence of learning rate scheduler
41
A3 Datasets used for zero-shot and linear probing
41
A4 Linear probing for CLIP (reimplementation) and CLOOB trained on YFCC
43
16

---

Under review as a conference paper at ICLR 2022
A.1 INFOLOOB vs. INFONCE
A.1.1 INFONCE: LOWER BOUND ON MUTUAL INFORMATION
We derive lower bound on the mutual information between random variables X and Y distributed according to p(..3) y). The mutual information [(X; ; Y) between random variables and
In p(æ, v) p(œ) p(y)
p( v) Ep(n,w) In p(x)
In ply x:) p(y)
I(X; Y) Ep(a..))
Ep(a,,)
(A1)
"InfoNCE" has been introduced in van den Oord al. (2018) and is multi-sample bound. In the setting introduced in van den Oord et al. (2018), we have an anchor sample given. For the anchor sample y we draw positive sample æ1 according p(æ) y)- Next, draw X (.2,,..,,.,, according p(X). which are n negative samples drawn iid according to p(z). We have drawn a set X {.11. I2,.... IN' } according to p(X y), which is one positive sample æ drawn by p(z, y) and negative samples {I2,.... EN } drawn iid according to p(æ).
The InfoNCE with probabilities is
x p(u Ei=1 p(y
IIafoNCE((, (X1; Y) = Ep(w) E
In
(A2)
where we inserted the factor & in contrast to the original version in van den Oord et al. (2018), where we followed Poole et al. (2019); Tschannen et al. (2019); Cheng et al. (2020); Chen et al. (2021).
The InfoNCE with score function f(x, y) is
f 3)) E
IntoNCE(X1 Y)
Ep(n) Ep(x/v)
In
(A3)
f(z,, y)
N
The InfoNCE with probabilities be rewritten
ply y xx1) Σ p(y xx.)
INfONCE (xroNCFFXI::: Y)
Ep(w)
Ep(x(m)
In
(A4)
p(ular))
Ep(()) Ep(x(1)
In
plu)
N Σ
plu (e:))
1=
p(y)
p( lu)
Ep(w) Ep(X
In
p(#1)
N
p(m:lu)
p(x.)
This is the InfoNCE with f(x, 9) = p(y) x))
Set of pairs. The InfoNCE be written in different setting Poole et al. (2019), which is used in most implementations. We sample N pairs independently from p(æ, y), which gives Z (( yi), (w2, (2),).).... (eN.yN), The InfoNCE is then
N
f( 9:) f(x;, y:
InfoNCE(X Y) = Ep(Xiv)
In
(AS)
N
t=
1=
17

---

Under review as a conference paper at ICLR 2022
Following van den Oord et al. (2018) we have
plyles mm)
[ Y) = Ex(w)
Ex(x(n)
In
(A6)
N
pl (m:))
i=1
p(y)
plec 1)) p(m1)
Ep(y)
Ep(xlv)
In
p(a.
2i=1
p(a,)
p(ac y)) (1=2p(m))
Fv((s)
F
In
+ In(N)
Σ i== p(ar; () lez
= Fp(y))
[pp())))
[Inp(i = 1 X, y)]l + In(N),
where p(i = 1 X, y) the probability that sample æ; the positive sample we know there exists exactly one positive sample in X.
The InfoNCE is a lower bound on the mutual information. The following incquality from van den Oord et al. (2018):
p( y)
I(X1: Y) = Ep(y)
Ep(mal1)) In
(A7)
p(x1)
p(ari)
-Ep(y)
Ep(an
p(æ1 (1y)
p(a1))
2 Ep(y)
Eptar b)
+
N
p(æ1 y)
Ep(r)
(x(1) (1))
In
+
p(x1)
Σ
N
p(a: y)
N
N p(x1 y)
pl x
p(æ, yn)
Ep(n)
Ex(X(1)
In
p(a1)
p(anly) p(Z))
pi lwl p(w.)
+ N 2im2
MinoNCE(X,, Y),
where the ">" is obtained by bounding In(1/N a) by In(a), which gives bound that is not very
tight, since a p(m]) can become small. However for the "ss" van den Oord et al. (2018) have to p(
assume
p(x; 18)
ply x2)
Σ
1,
(A8)
N
p(x:))
N
p(y)
which is unclear how to ensure.
For proof of this bound Poole al. (2019).
We assumed that for the anchor sample y positive sample æ has been drawn according to p(æ) y).
A set X {Iz...., xx of negative samples is drawn according to p(æ). Therefore, we have set X (2,, xx } that is drawn with one positive sample œ and N negative samples
X (x2,..., J. We have
p(X)
I ((x.)
(A9)
1=2
p(X y) p(#1 y)
Il p(x))
(A10)
p(X) I p(x)) =
(A11)
18

---

Under review as a conference paper at ICLR 2022
Next, we present theorem that shows this bound, where largely follow Poole et al. (2019) in the proof. In contrast to Poole et al. (2019), we do not use the NWJ bound Nguyen al. (2010). The mutual information is
In p(a1 I y) p(x)1)
)]
T(X1: Y) Expa... w))
(A12)
Theorem A1 (InfoNCE lower bound). InfoNCE with score function f(x, y) according to Eg. (A3) is lower bound the mutual information.
f y) EE
1(X1: Y)
Ep(x)r(Xip)
In
InnfONC((x Y).
(A13)
f(x;; y)
N
InfoNCE with probabilittis according to Eq. (A2) is a lower bound on the mutual information.
)
ply x
1(x1; ; Y) 2 Ep(e)p(X1e) In
Ei= p xi)
IintNNC((X1 Y).
(A14)
The second bound Eq. (A14) is a special case of the first bound Eq. (A13).
Proof. Part (I): Lower bound with score function f(x, y)-
For each set X = {æ:..... x. J. we define as data-dependent (depending on X) score function g( X) that is based on the score function f(x, y). Therefore we have for each X different data-dependent score function y based on f. We will derive bound on the InfoNCE, which the expectation of a lower bond on the mutual information over the score functions. For score function g( y. X), we define variational distribution g X) over æ
<(<1 y, 13,8) 8) p( g(x y, X)
(A15)
Xyy
Z X), = Ep(a)) g((11, y, x))
(A16)
which ensures
/ g(<1 8) dx = 1.
(A17)
We have
g(x1 X) p(x))
g( y. X) Z(y,X)
(A18)
For the function g, we set
f(x1, y Σm f(x,, y)
,00,,,,88)
(A19)
1
N
For the function f we use
f(x11 v) = exp(7-1 sim(21) y))
(A20)
where sim(x, y) is typically the cosine similarity.
We next show that InfoNCE is lower bound on the mutual information.
19

---

Under review as a conference paper at ICLR 2022
In p(a0 y)
1(X1; V) = P)(X) [(X1: ; Y)] = PBC))
Pecasse) 201
(A21)
p(x1))
p
y)
a(a0) y. X)
P(8)
F p)()1.1)
g(0)
X)
p(a11)
g(01 y. x)
KL(p(x1 y) ll g y, x))]
E ()(
+ Ey(w))
p(x1)
g( yy, X)
Ep(ar-v) In g(x1.3,11 yr
2 (X)
E Ep(21.2)
E Ex(x)
p(æ11
Z(y,
E p(()
Ep(a).0) v))
In (1) y, X)
In (ptan)) g y, ".,,)
= P)())
Ep(r) Ep(male) lm)
Ing(211 y, x)] - In Epp(ma)) [0(21, y, x3) x)]]]
Ing(an, y, x)] {]]]] - E Fx(x) Fip(s) inn (Eecee)) g(( v, x) x]]]]]
P)(
Fp(w))
F Fypcar (x)
2 Ep(w)x(X|x)
In ( y, xx]
- Exx() Ep(()
Ep(an) )
0(111,3,70 -1]]
f(x11.
f(011 .V)
Ex(y)x(Xin)
In N Eim f(xr,, 3)
Ep(y)
Ep(x)
Em= f( 3))
N
In f(11.3)
1
f y)
(x(v)
Ep(y)
Ep(x)
NV E f(zv, y))
N
Ei=l f(at 3)
i=1
N
In f(1188)
Σ i=1
f(Zi, 8)
(
Ep(n)
Ep(x)
E f(x,,
Σ f(zi, 33)
f(x1, y)
(x(v)
In N Lizl f(x1,))
LInGNCE(X1 Y).
For the first "2" we used that the Kullback-Leibler divergence is non-negative. For the second "Z" we used the inequality In a & a for 0.
Part (II): Lower bound with probabilities.
the score function f is
f(x. y) = p(y) x),
(A22)
then the bound
-mm f(x;, wn )
/ [(x y)
Ep(n)((X/) In ply x))
1(X1 : Y) 2 Eptn)x(il))
In
ΣN p( xx))
(A23)
p(y)(r)) p(y)
Ep(n)e(Xlp)
In
I Y).
N ply mc)
N
-=
p(U)
This is the bound with probabilities in the theorem.
20

---

Under review as a conference paper at ICLR 2022
A.1.2 INFOLOOB: UPPER BOUND ON MUTUAL INFORMATION
We derive an upper bound on the mutual information between random variables X and Y distributed according to p(æ,3) y). The mutual information I(X; ; Y) between random variables X and Y is
In p( y) p(2:) p(y)
In p( y)
Ep(a.w) In p(y x) ply)
I(X: : Y) Ep(a.a)
Ep(a.))
p(zz)
(A24)
In Poole et al. (2019) Eq. (13) introduces a variational upper bound on the mutual information, which has been called "Leave one out upper bound" (called "L1Out" in Cheng et al. (2020)). For simplicity, we call this bound "InfoLOOB", where LOOB is an acronym for "Leave One Out Bound". In contrast InfoNCE, InfoLOOB is upper bound the mutual information. InfoLOOB is analog to InfoNCE except that the negative samples do not contain a positive sample. Fig. 1 and Fig. 2 in Cheng et al. (2020) both show that InfoLOOB is a better estimator for the mutual information than InfoNCE (van den Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010).
The InfoLOOB with score function f y) defined as
f y) Σ
IIntoLOOB (X1 Y) Ep(e))
In
(A25)
sass y)
N-1
The InfoLOOO with probabilities defined as IInfot.OO((x, 1 Y) Fip(s) Fxcx v)
plu x
In
(A26)
Σ ply x0:)
N-I
This is the InfoLOOB with [ar y) = p(y xt).
The InfoLOOB with probabilities can be written in different forms:
ply xi)
[ ; Y)
Ep(n)
Ex(x)))
In
(A27)
Σ i= p(y
xii)
N-I
p(1)))))
plan lwv plan )
Fpp(u)
Fep(xin)
In
((y)
Eep(n)
Fx(X(T)
In
N-1
1
Ei=2
p(vla,) p(y)
1
Di=2
p(a. (2) p(z.)
Set of pairs. The InfoLOOB can we written in a different setting (Poole et al., 2019), which will be used in our implementations. We sample N pairs independently from p(z, y), which gives X {x11 y1), (22, (xN, UN)}. The InfoLOOB is then
N
f( y1) Ein [w 9:)
InfoL.OOn( Y) = (p((())
Σ i=1
In
(A28)
N
N-T
We assume that an anchor sample y is given. For the anchor sample y we draw a positive sample 21 according to p(a, y). Next, we draw set X (ac...... .NN of negative samples according to p(x y). For a given y, the æ that have a large p(æ y) are drawn with a lower probability p(æ y) compared to random drawing via p(a)). The negatives are indeed negatives. We have drawn first anchor sample y and then X {xI,.... xx J. where is drawn according to p(æ y))
and X {x N are drawn iid according to p(x yy. We have
II 1=2
y)
(A29)
N II
p(X y) = p(#1 11)
p(x v)
(A30)
1=2
p(X v) p(xx) p(x1)
p(x, y)
(A31)
21

---

Under review as a conference paper at ICLR 2022
We assume for score function f(x
VyAa 0 < f(x,
(A32)
We ensure this by using for score function
f(z y) exp(r sim(x,y)) where sim( y) is typically the cosine similarity.
(A33)
InfoLOOB with score function fx, y) is
InnoLLOO((X Y) = Ep(x) Ey(xIw)
In
f(z1. y)
(A34)
E f(x y)
N-1
The reference constant Zy) gives the average score f(x y). if the negatives for y are selected with lower probability via p(x y) than with random drawing according to p(x)).
Z(y) Eptetv) [f(x,)).
(A35)
We define the variational distribution
< v) = p(æc) f(x, v)
(1) Ep(a)) [f(x, y)]
(A36)
With the variational distribution g( y), express our main assumption. The main assumption for the bound is:
Ep(r) [KL(p(> y) || q( v))]
< Epts) [In (1) - In Z(v)l
(A37)
This assumption can be written as
ply x)) Z(y)
p(y) f(x, 3) }]
Ep(r)
Ep(zlyy
In
< 0.
(A38)
This assumption ensures that the x with large p(x y)) are selected with lower probability via p(x y) than with random drawing according to p(æ:). The negatives are ensured to be real negatives, that is, p( 3) is small and so is f(r, y). Consequently, we make sure that draw with sufficient small f(x. y). The Kullback-Leibler gives the minimal required gap between drawing [ax y) via p(æ) and drawing [(x y) via p(a: y).
EXAMPLE. With hyy) > 0, we consider the setting f(x, = p(y a:) h(y)
(A39)
p(y)
p(x) p(y)
p(y x)) h((3) p(y)
7( (1) = h(y) ply x) C(v)
Cly) = Ep(ar)
(A40)
The main assumption becomes
In Z(y)
Ep(r)
Ep(aolu)
< 0.
(A41)
h(y))
The main assumption holds since
p(y x) h(v)
p(x) p(y)
ply x)) h(y)
Z(y) Ep(mly)
dx
(A42)
p(y)
h(y) p(y x)) C(y)
p(y)
= - p(zx) C(y) -1 dx = C(y)- (Excee) (
(ply a)) h(y) -1)
p(v)
1-) -1
p(y x)) h(y)
p(y x)) h(z)
Epp(a)
-Ep())
p(y)
p(y)
ply, x)) h(y) dx h(v)
p(y)
22

---

Under review as a conference paper at ICLR 2022
where used for the < Jensen's inequality with the function f(a) 1/a, which is convex for a >>
For score function f(x, y) and distribution p(æ y) for sampling the negative samples, have defined:
Z(y) Ep(al)) ( [<xyy))
(A43)
(y) Ep(t) [f y))
(A44)
( y) p(æ) f(x, y)
(A45)
(²)
Next theorem gives the upper bound of the InfoLOOB on the mutual information, which is
T(X1: ; Y) Epter.0 w) In p(ar y)
(A46)
p(x11)
Theorem A2 (InfoLOOB upper bound). If (x2..... IN} are drawn iid according to p(æ) y) and if the main assumption holds:
Ep(w) [KL(p(> () || g(x yy)] < Epiz) (n)) In Z(y)l
(A47)
Then InfoLOOB with score function f(x,, y) as in Eq. (A25) is an upper bound on the mutual information:
f f(r1, y) En f(x,, v)
)[
I(X1 Y) < Ep(r) Eppx(1)
In
InfoLoo (X1 Y). (A48)
1 N-1
the negative samples X fx2...., ØN } are drawn iid according p(x), then InfoLOOB with probabilities according to Eq. (A26) is an upper bound on the mutual information:
Σ ply X,) )
pΦy x1) N-I
I(X1; Y) & Epin) Ex(X(e)
= JiafLLOB (X1 Y) (A49)
The second bound Eq. (A49) is a special case of the first bound Eq. (A48).
Proof. Part (I): Upper bound with score function f(x, y)-
23

---

Under review as a conference paper at ICLR 2022
In p y)
I(X; Y) pp(x).0)
(A50)
p(x1)
In p(
1)
g(<1 1)
9
p(x1)
In ( y)
E)(m1.))
+ Ep(r) [KL(p(æ)) 3) g(x) b))l
p(x1)
In ( y)
F))
Ppp(e) IIn Po(as) (f(11, v)] In 7(v))
p(x1)
In (( y)
Fp(a.) [f v)
w))
In
p(x1)
(zy)
In f(21.3) 3)
Ep(a:) ) [f b)]
v))
[f(11, y)]
(
In fx1 y)
= M)(
(ly)
f(x1,3)
= M)
In E
Dizz f(x,, 11)
N-T
Ep((ra) [In f(x1,
-Ep(s)
In
Eppx(o)
Σ f(z y)
N
< Ep(er. [In f(21, b)1 - Ep(w)
Ex(X(s)
In
f v)
N -
in
[x) y)
= Pp(())
N En f(x;;,)) v)
InnoLLOO((X Y).
where the first "<" uses assumption Eq. (A37), while Jensens's inequality was used for the second "<" by exchanging the expectation and the "In". We also used
N Z(y) = Z(y)
Ep(x1n)
f(x,, y)
Epces vv) [S v)1
N
N
N
t=
(A51)
Part (II): Upper bound with probabilities. If the score function f is
f(x y) -p(y a:)
(A52)
and
p(x () = p(x))
(A53)
then
P(X y) p(X 1)
(A54)
7(y) Fxp(a) (p(y x)) p(y)
(A55)
z²(y) Ep(z) (p(y x)] p(y)
(A56)
q y) p(x) p(u x)) p( y)
(A57)
ply)
KL(p(= 3) || q(x v)) KL(p(æ v) p(x v)) 0.
(A58)
24

---

Under review as a conference paper at ICLR 2022
Therefore, the main assumption holds, since
0 Ep(w) [KL(p(
y) ll g(x y))]
Eplu) [In Zn
In [(y))
(A59)
The bound becomes
p(y 21)
1(X1: Y) < Ep(w)
E Ex(x( In
(A60)
1
N
p(y x.))
i=
p(a (mn)
Exwn
Ex(X(()
In
pla)
IntLOOB (X1: Y)
p(u )m:) PLM)
1==
An alternative proof is follows:
N
p(y)
I(11; Y) =[ Y) Ep(y)
(A61)
N
p(y)
2
1
N
plu axa)
= I[(x Y) - Ep(w)
In Bp(x(
- N 2m
p(y)
1
p(y (x4))
< I(X1; Y) Ex(w)
Ex(X(s) In
N
p(y)
N
p y)
p( y) p(x;)
Fp((1)
In
Fop(v)
Ex(((1)
In
p(x1)
N
1=2
p(a la)) p(an)
= Ep(y)
Ep(X(v) In
pla. p(a,)
Innfot.OO Y)
where we applied Jensens's inequality for the exchanging the expectation and the "In" obtain the incquality.
Experiments that compare upper and lower bounds as mutual information estimates are provided in Cheng et al. (2020) and in Poole et al. (2019). In Fig. 2 in Cheng et al. (2020) it is shown that InfoLOOB a good estimator of the mutual information.
A.1.3 INFOLOOB: ANALYSIS THE OBJECTIVE
This subsection justifies the maximization of the InfoLOOB bound for contrastive learning. Maxi- mizing the InfoLOOB bound is intuitive as was introduced as an upper bound on the mutual information in the previous subsection. Still maximizing the InfoLOOB bound leads to a good approximation of the mutual information, in particular for high mutual information.
InfoLOOB with neural network as a scoring function is not an upper bound on the mutual in- formation when not mnder-sampiing As we use InfoLOOB on training data for which we do not know the sampling procedure, we cannot assume under-sampling. Therefore. we claborate more on the rationale behind the maximization of the InfoL.OOB bound. (I) We show that InfoLOOO with neural networks as scoring function is bounded from above. Therefore, there exists maximum and the optimization problem is well defined. (II) We show that InfoLOOB with neural networks as scoring function differs by two terms the mutual information. The first term is the Kullback-Leibler divergence between the variational ((x y) and the posterior p(x y). This divergence is minimal for q( y) p(= y), which implies f(y x p(y x). The second term is governed by the difference between the mean E (x. y)] and the empirical mean 1/(N - 1) Σ; f(x, y). Hoeffding's inequality bounds this difference
demonstrate in this subsection. Therefore, the second term
25

---

Under review as a conference paper at ICLR 2022
is negligible for large N. In contrast, the KL term is dominant and the relevant term, therefore maximizing InfoLOOB leads fz z) R p(y x).
We assume that anchor sample y is given. For the anchor sample y, draw positive sample æ1 according to p(æ, y). We define the set X {02,... ......., ,N, of negative samples, which are drawn iid according to p(z). We define the set X [1..... ØN
We have
x(8) II p(x:),
(A62)
N
p(X y) p( y) II p(x.) = p( 1 3) p(X).
(A63)
1=2
P(X) p(x;) p(x1)) p(X).
(A64)
We use the score function
f(x,y) cxp(r-1 sam(x: g))
(A65)
where sim(x, y) is typically the cosine similarity.
The InfoLOOB with score function f(x y) defined as
(xx 9)
Inaoso.OB(ii Y) Ep((o)
Fx(X(0)
In
(A66)
N-I Em
[ y)
We define the variational distribution
g 4) p(x) f(x, y)
(A67)
Z(y)
Z(y) = Ep(() [f y)
(A68)
26

---

Under review as a conference paper at ICLR 2022
The next inequality shows the relation between I(X, ; Y) and ( Y) for random variables X1 and Y.
p( y)
1(x1 : Y) M)(
In
(A69)
p(x1)
In p
y) q(001 y)
M)(
((11
y)
p(a1)
In (an [y)
Fxp(w) [KT(p( yy) g(x) ())]]
p(x1)
In f(x1,1))
M)(
Ep(z) [KL(p( v) || (( ())]
7(y)
)1
((x1
() In Epp(11)
+ Expc) [KT(p(a () g(<; [y)]]
N-1 Eiz s(x,,))) v)
Ep(an.) [In ((r1,)) v)] Ep(x) In Ep(x(T)
1
(ae y)
N
i=2
+ Ep(n) [KL(p( y) |l g(<1) B))]
1
Ep(aaa) In In ((11,))) Ep(u) Ex(X(1) In
f f y)
N
i=2
]]
N
f( y) )
+ +Ep())
Bx(((1)
In
f( y)
Ex(cn)
In
Σ
N
1=2
Ep(n)
[p(xp(>1
1) Ep(X(1) In
g(<11 B))
f(x y)
Ep(())
+ Epp()) Ep(x(1) In
Σ
(ats y)
N-1 Σin (a y)
N
im2
f y) )]
Epty)
In Ep(x(l)
+ Ep(w) [KL(x(a01 1/) g( I y))
N
(X1 Y
]1 Ep(n) In Ex(X(en)
N
f f(a v) 1]
Eptn)
Ex(((1) In
1
Σ
f(a;, y)
Σ
N-1
N
i=2
2=
[KL(p( Φ1 + Epiz)
y) q((x1
y))]
OInfoLOO X1 Y)
]] Epin) [In (Eep(ars (f g)i)
1
Epis)
Ex(x(v) In
Σ
fax y)
N
2
+ Eptn)
[K[ x1
v) q(<1 v))
InfOLLOBB X1 Y)
Ep(za) [f (x 4)1
Ep(n)
E Bxxx)
In
N-I Σm f( 8)
+ EPt()) [KT.(p(=) v) |l q(<1) vi)]
LittoLOB((Xi Y) - DE + Ep(x) [KL(p(æ1 1) g(æ) bv)]]
where we used
Ep(mn) [f(21, y)l DE Ep(x) Ep(x) ln
(A70)
N-1 EN, f(x,, y)
27

---

Under review as a conference paper at ICLR 2022
and
N f y)
Z(y) Ep(as) [f(21.8)) = Ex(x)
(A71)
N
Ep(Xlv)
f(z;, y)
N -
2=
Since both KT. and DE are non-negative (for DE see below), to increase InfoL.OOB we have either decrease KT. increase DE.
Bounding DE. Next we bound DE. We define
L x" B
In 23
(A72)
The log-sum-exponentie (Ise) is the maximum of L on the N-dimensional simplex D with D {2
Σ, x = 1,0 & 2 (Gao & Pavel, 2017):
1so(3,x)
max x xx 8 =ED
24 In 2g
(A73)
For some E D) we have
N
Es (Ise(B,a)) 2 Ea
T a - 8-1 1
Σ
2,
In 21
xx Ea [a] - 8-11
24 In 2g
(A74)
i=1
therefore
N
Ea [Ise(B, a)] 2 max zfE Ea [a]
-8-1
Σ [=1
2i In 1se(3, Ea {a))
(A75)
sED
We obtain
exp(r sim ( 21)
Ep(n)
Ep(x)
In Ep((m)
(A76)
N Σ i=2 exp(r-1
sim Iz
Ep(n)
In Ep(zs)
(exp(5 -1 sim 11 y))] In N exp(r Eppra)) [sim (z,, v)))
Ep(y)
[In Ep(mn) (exp(r sim x1, x1))))] T -1 Ep(m1) [sim(x1 (x1 y)]]
We obtain via Jensen's 's inequality
exp(r- 1 2))
Ep(n)
Exex) In Ep(ee)
(A77)
Di=2 exp(r-1 sim (z,, 3))
N
N In Ep(ar)) (exp(r -1 sim(x1, f))l In N - 1
2 Ep(y)
Ep(m))) [exp(r-1 sim (am(z1,))) (I1, (z1.8))] f))]
0.
If combine both previous inequalities, obtain
0 < DE
Ep(w) [I Ep(ar) [exp(r-1 sim( ())] -1 Ep(n)) [sim(x), 8)] (A78)
28

---

Under review as a conference paper at ICLR 2022
In partieular, for bounded (im( y), we get
DE
< r-
max inaxsimmay (21 y) - min sim (21, y)
(A79)
9121
while Hoeffding s lemma gives
0 << DE
E 8
T
max sim v) - min sim((1 v) y.m
(A80)
y.
Thus, for bounded sim ( y). DE is bounded, therefore also InfoLOOB, For sub-exponential distributions with variance o², for which Bernstein's 's condition with > b holds (Eq, (2. in Wainwright (2019)), we get (Proposition 2.3 Wainwright (2019)):
o²
E DE - bT)
(A81)
-
Next, show that DE small. Hoeffding's inequality states that if (ax, y) [a, b] then
p
(olce [f v))
sax v) c)
2 exp (-
2 (N - 1) )
-0-a² a)2
(A82)
N
i=2
For
N Σ
Ep((s) [f(21,))]
f(z;, y) &
(A83)
N-1
1=2
we have
Ep(x:) (()01,))]
D 2 [wi y) +
<
N-1
(A84)
N-I Ei= (ar,,y)
N-I Ei [(x;,)]
N-1 Lime f(x,,)) y)
Z -
where we used In & 1 for 0 < a. Analog for
f( v)
Ep(r)) [f(21,8)]
(A85)
we have
Epter) [[(
Ep(e)) [[
In
2 In N-1 Eiz f(0 y1)
(A86)
Epp(r) (x +
-- Ep(ee) (s y)] + 6)
Ep(ma)) [f ( y)]
Ep(m)) [f(21,8)
Z
where we used - aaa 1 for 0 < In summary, for
N N 1
Ep(m)) (f(21,
Σ
f y) <
(A87)
we have
E Ep(2y) (f
E
In
< --6
(A88)
7
N-1 ΣN, f(x;, y)
29

---

Under review as a conference paper at ICLR 2022
It follows that
E 7
DE <
(A89)
7
DE averages the In-term over y and X. therefore it has an even smaller bound than the bound above on the In-term. Consequently, for small b and large N, the term DE is small.
KL is decreased by making the variation distribution ( y) more similar to the posterior p( y). The value DE only depends the marginal distributions p(y) and p(æ). since p(X) I p(æ,). The value DE can be changed by adding an offset to fa: 3). However, scaling f(x y) by a factor does not change DE. Consequently, DE is difficult to change.
Therefore, increasing InfoLOOB is most effective by making g( y) more similar to the posterior p(x1 y)-
Gradient of InfoLOOB expressed by gradients of KL and DE. Assume that the similarity is parametrized by giving sim (ar, y; w))
KL(p(æ) y) ll g(ω, b)) = / p(æ01 (v) In p(x)) y) ((x1 y)
dx
(A90)
= - / plz) y) sim(æ1, w) dæ + + C
where C' is independent of w.
Next, we compute the derivative of KL with respect to parameters w.
ØKL dw
(A91)
Osin( yi w)
1 / p(a))) Osim(21, y/: w)
exp(r-1 sim y: w)) Osim (#1, yı; w)
p(a1 y)
day
day
Ow Osim (I1, y/ w)
Z
Ow
/ p( lg)
-1 / p(20))
exp(r- sim 1 y: w))
Osim (x1, y; w)
day
dæ
tw
Ou
-- / p(z lt) Osim (aim(x1,15 yi w)
dx --1 / g(20) (t)
Osim Daim((111 yi w) dæ
dw
Ow
T-1 / (g(211 (g) - p(x1 y))
asim( y: w)
dx1
Ow
The derivative is the average difference between the posterior distribution p(æ1 | v)) and the variational distribution q(E, y) multiplied by the derivative of the similarity function. both distribution match, then the derivative vanishes.
30

---

Under review as a conference paper at ICLR 2022
Next, compute the derivative of DE with respect parameters
ODE
(A92)
dw
Oln
Zi=2 r-1 exp(r- sim(a,, y: w))
Osim (as (y:
-E((y)
Ep(v)
Bx(x)
N-1
Ero
N-1 ΣØ= f(zg.
x- /
Osim((X) [11, y: w)
Ep(uu)
q( 11)
dæ
Er
N-1 Eiz exp(T -1 sim ( y: w))
(sim(an; (3:3)))))))))
Eppt))
Bx(X)
N-I Ex=2 f(zg, v)
=r- Ep(n) f g(211 3)
Usim y: w) dx
dw
N
f(z,, y)
Osim( y: w) Ow
--1 Ep(r)
E Ex(8) N -
Σ
1=2 N-1 Σj=2 f(2,, y)
p(a1)) 1(x1,3)
Øsim( y: w) dan
-- Ep(w)
Epp(c)) [f( y)]
Er
-1 Ep((r)
F x(X)
1
Σ
f(z,, y)
Osim( (, y: w) dw
N
N-1 Σド= f(z).y)
f y)
Ósim(ar, y. w)
--1 Fep(2)
E Ep((a))
Em((r) [f y)]
Ouc
1
f(x,, y) N-1 Enz (a)
Osim( a y; w)
r Ep(n)
E ((X)
N 1=2
Ow
1
f(x;, 8)
Osim( (z, y: w) Ow Osim (2.; y; w) Ow
---1 Epp())
E(p(ar)
N
Ep(()) [s y)]
N Σ
[ y)
-1 Ep(w)
E p(X)
N
N-1 Σ5= f(z,,8)
1=2
f( y)
Osim (2 y w) Ow
-- Ep(w)
E (()
N
Pxp(ae) [f y)]
i=2
1
f( y)
Osim( (2, y; w) Ou
- Ep(x)
E ((()
N
Σラ=2 f(æ),8)
Ósim(ar, y; u))
-r- Ep(()
-1EEp())
E (())
Σ
f f(x, y)
N - 2=
Ey(n) [f(
N-1 Enz fa,
1
Ow
- -7-1Epoi Ep(w)
Ex(X)
1
1
f(z,, y)
Ōsim (zy y/; w)
N
En f(æ,, (1 N-1
1
dw
2=2
The derivative is the average of
xー (a) multiplied by the score function and the
derivative of the similarity function. The average is over y and X, therefore the whole derivative becomes even smaller. Consequently, for small a and large N, the derivative of DE is small.
Note that for
Ep(a,) [f(11,))]
f(x y)
E
(A93)
N
2=
31

---

Under review as a conference paper at ICLR 2022
have
1
1
(A94)
Z
1
Ej=2 f(zj. y)
Z
Z +
Z(Z +c)
N-1
1
(A95)
Z
EØ=2 f(
Z
Z
-
Z(Z cc)
N-1
therefore
< Z(Z - c)
(A96)
Z
N-I Ej- fax y)
the expectation well approximated by the average N-I Ej=2 J (zz y), then both DE and its gradient are small.
Derivative of InfoLOOB via KL and DE:
Ol (X1 Y) ODE
0KL
(A97)
On
Ow
In this gradient, the KT. term is dominating, therefore (ar, y) is pushed to approximate the conditional probability p3y æ). Modern Hopfield networks lead to larger values of p(y x) as the mutual information becomes larger, therefore modern Hopfield networks help to push f(x, y) to large values. Furthermore, modern Hopfield networks increase Z, which is in the denominator of the bound on DE and its derivative.
A.1.4 INFONCE AND INFOLOOB: GRADIENTS
We consider the InfoNCE and the InfoLOOB loss function. For computing the loss function, we sample N pairs independently from p(a, y), which gives the training set { y1), (x2, y2)...., (xN.yN)) InfoNCE and InfoLOOB only differ in using the positive example in the negatives. More precisely, InfoNCE uses for the matrix of negative samples X (x11.... IN), while InfoLOOB uses = INN).
InfoNCE.
The InfoNCE loss
N
1
[x Wi) Σラ I (at
LIfaNCE
In
(A98)
N
1 N
N
1=1
where we used
f( yi)
LInfONCE(y:) = In
(A99)
f= f(x 31.)
For the score function f(x, y), we use
f(x, v) exp(r- sim ())
(A100)
sim (x, y) y
(A101)
with the temperature.
The loss function for this score function is
LInoNCE(y)) -- TT x1 -- Ise (7 (7-1,XT XTx)
(A102)
where Ise is the hog-sum-exy function (lse):
1se(B, a) 3 Σ exp(Ba:)
(A103)
32

---

Under review as a conference paper at ICLR 2022
for 3 >> 0 and vector (a1..... ".l....;00 aN).
The gradient with respect to y is
(LIntoNCE()) T æ1 - X softmax (r y)
(A104)
dy
which is the positive example 2 that fits to the anchor example minus the Hopfield network update with state pattern y and stored patterns X and then this difference multiplied by r-
This gradient can be simplified, since the positive example is also in the negative examples. Using p = (P1..... pPN softmax (r X y). we obtain
ALLSONCE
(A105)
dy
= (1 - p1)
21 P-P1
1
X (softmax (T-1x²') - (01, (pI.0....,.))))) .)),)
(LISOLOOB(()
-1 1 (1 pı)
X softmax (F-IXTy)) = (1 pi)
Ø1
By
where
X (softmax (T-1x" (p), 0. 0).)..))7)
(A106)
1 P1
1
X PPP..P2....... (p1, 0, 0)²)
P1
1
X(.. PNN
ps x
P1
P1
the softmax average over the negatives x; for 2 < N without æ1. It can be easily seen that
I-pi Σ 1-m1
T-p: 1. the derivative of the InfoLOOB see below.
The gradient with respect to 21 is
(LLioNCC(()) dx
exp(T x y)
--1 y +
Ei=1 exp(e
N
y
(A107)
pt) y
(A (A108)
Consequently, the learning rate is scaled by (1 p1). The sum of gradients with respect to x and x; is
ALimONNC (3)
N LLmtONCE (8) - + -- y =oooftmax (x-
(A109)
dx
dæ:
1=1
0=
where 1 is the vector with ones. However, the derivatives with respect to the weights are not zero since the x; are differently computed.
InfoLOOB.
The InfoLOOB loss is
N
f(x;, 3/:)
N N
LnnfoLOOB =
In
=
Lg.) (A110)
N
Σ == 1i f v.)
N-1
where we used
f( y.)
LinfoLooa(y.) (3:)
In
(A111)
N-1 -= f(x y.)
33

---

Under review as a conference paper at ICLR 2022
For the score function f(z, y), we use
f (æz v) exp(r sam y))
(A112)
sim xz y) y²
(A113)
with the temperature.
The loss function for this score function is
[InfoLOOB(U) - - yo x1 + r- Ise (x,),,')
(A114)
where Ise is the The gradient with respect to y is
function.
(LittLOOBB()) dy
+ X softmax (f
(A115)
which is the positive example æ that fits to the anchor example minus the Hopfield network update with state pattern and stored patterns X and then this difference multiplied by r-
The gradient with respect 1
ALutss.onn())
- y
(A116)
dx
The sum of gradients with respect to æ and æ; is
O. UmmoLOOO (y)
Σ
--1 y + --1 y 1 softmax (r' 4)
0x.
(A117)
where 1 is the vector with ones. However, the derivatives with respect to the weights not since the x; are differently computed.
Gradients with respect to --
The gradient of the InfoNCE loss Eq. (A98) using the similarity Eq. (A100) with respect to 7 is
aL
yh æs + T X softmax (r x y)
(A118)
0r-1
y (x: X softmax (r x y))
(A119)
which is the similarity of the anchor y with the difference of the positive example æ; and the Hopfield network update with state pattern y and stored patterns X. The gradient of the InfoLOOB loss Eq. (A110) using the similarity Eq. (A112) with respect to -1
θr = - y² σ1 + 8" softmax
(-
(A120)
= - nr (x1 X softmax
x
(A121)
with the difference that the Hopfield network update done with stored patterns X instead of X.
X softmax (oaaaxx(-'""" (--1XTT)
Without the positive example æi in the stored patterns X, the term 21
in Eq. (A 120) will not decrease like the term æ X softmax (7-1x" in Eq. (A118) but grow even larger with better separation of the positive and negative examples.
A.1.5 INFOLOOB AND INFONCE: PROBABILITY ESTIMATORS
In McAllester & Stratos (2018; 2020) it was shown that estimators of the mutual information by lower bounds have problems they with serious statistical limitations. Statistically more justified for
34

---

Under review as a conference paper at ICLR 2022
representing the mutual information is difference of entropies, which estimated by minimizing the cross-entropy loss. Both InfoNCE and InfoLOOB losses can be viewed as cross-entropy losses.
We sample N pairs independently from p(x,y), which gives Z
{(0)) y1), (Z2, y2)..... yN)]. We set X { [11 #2...., NN and Y = {31, 92......................... yn so that, Z X x Y. The score function f(x, y) is an estimator for p(æ: y). Then obtain estimators 9 for the conditional probabilities. ((y: z; Y', (y:H) is an estimator for ply; x.)) and
((r; yi: X \ {2;)) estimator for p(æ; y.). Each estimator ④ uses beyond (æt, ys) additional samples to estimate the normalizing constant. For InfoNCE these estimators are
Mat 91)
[a yi)
G' (y: 34 Y (311)
(A122)
N Σソ= f(z)
Ep(() [ y)
/ (1 y/)
(a; yi)
(r, yk, X (ai))
(A123)
Σÿ= (ang y:)
Ex(x) [f ys)l
N
The eross-entropy losses for the InfoNCE estimators are
N
f(ai, yi)
LI
ln
(A124)
N
2i=1 f(z;
[=l
N
N
fa Us)
LILENCE
In
(A125)
Σラ=
N
farg vi)
For InfoLOOB these estimators are
f(x;, Bi)
f( Mi)
(y1 2b, Y (3,tt)
(A126)
N-1 Σ =11
f( Ys) [x yi)
Epin) (f y)]
[wi yi)
i yi, X (a:))
(A127)
N-I Σj-1sti [2 yi)
Ep((n) [fx y:)]
The crosspyntropy losses for the InfoLOOB estimators are
N
f(x,, yi)
L I
In
(A128)
ΣØ 1isti / (I vy)
N
N-1
N
(xi Vi)
2 L
=
In
(A129)
N
Σ ビj=1,j f (I) Mi)
==1
N-1
The InfoLOOB estimator uses for normalization
N Σ
Ep(a)) [f
y:)l
(A130)
N
j=
N Σ
1
Eptn) [f(2r,))
f(,,,)))
(A131)
N
j=1,fi
in contrast to InfoNCE, which uses
N
Ep(a) [f
f (æj, y1)
(A132)
N
N
Ep(n) [f
Σ
f(x,, (a)
(A133)
N
If InfoNCE estimates the normalizing constant separately, then it would be biased. (æ,, y1) is drawn according to p(z,, y:) instead of ((z:))(())) In contrast, if InfoLOOB estimated the normalizing constant separately, then it would be unbiased.
35

---

Under review as a conference paper at ICLR 2022
A.1.6 INFOLOOB AND INFONCE: LOSSES
We have N pairs drawn iid from p(æ, y), where we assume that pair (æ; yi) is already an embedding of the original drawn pair. These build up the embedding training set Z {(x11 y1), (Iz, (2)..y2)..... y2)..... (IN, yN)} that allows to construct the matrices X (I1, 22,,...... zN) of N embedding samples n, and Y (y), (31,,/,,... ./.............................. ... of N embedding samples yı. We also have M stored patterns U (u),..., u,r) and K stored patterns = (vi,...,
The state vectors æ; and y/ are the queries for the Hopfield networks, which retrieve some vectors from U or V. We normalize vectors |a:]| = =||yi|| = | || = ||0:]| =. The following vectors are retrieved from modern Hopfield networks (Ramsauer et al., 2021):
2. U softmaxx(3 U²x;), Un: U softmax(3 UT y.),
(A134)
V2, = V softmax(3 ( V xxx;). Vu = V softmax(3 VT
(A135)
where U2. denotes an image-retrieved image embedding, Uy: a text-retrieved image embedding, Va, an image-retrieved text embedding and vy, text-retrieved text embedding. The hyperparameter corresponds to the inverse temperature: 8 = retrieves the average of the stored pattern, while large B retrieve the stored pattern that is most similar to the state pattern (query).
We consider the loss functions
exp(T x.551 y:)
exp(T x y1) jj=1= exp( 7- x y/)))
LInfONCE
log
log
N
exp(p- 2!! yy)
N
(A136)
N
exp(T- x.. x. 4/))
N
cxp(r -1 xx.y1) yi)
LInoLLOB
log
log
jji exp(T x.. 4/))
N
jæi exp(T 1 x² y:)
i=1
i=1
(A137)
N
exp(T -1 U.. Vy.)) log
N
exp(T- U ) log
H-UVUV
m.
InfoLOOB
N
Σ i cxp(T-1 UT, V,,V,,
N
j$ exp(r- UT V2.)
1=1
1=1
(A138)
N
exp(T- -1 UT
1
N
exp(T
VT.VV
-UUVVV
log
log
y/1
InfoLOOB
871 exp(T-) U2 UT
N
# exp(T
VT Vg, (A139)
where for InfoLOOB the sum Ejer jes in the denominator contains only negative examples j. We do not consider the loss function elaborated in the following.
L H-VUV InfoLLOB because of the high variance in the dot product U V,, as
Let us consider the dot product between the anchor retrieval with the positive pattern retrieval for the loss functions with Hopfield. In the first term of the loss function Eq. (A138), Uz: is the anchor with V,, the positive sample and Vy: with Ua, the positive sample for the second term, since the anchor also appears in each term of the denominator. Equivalently the same is valid for Eq. (A139), but with positive samples Vz, and y. respectively. These dot products can be written as
U m;
Vy. = softmax (3 x" U'V softmax(5 (3 V VT y:)
(A140)
U U y.
softmax (3 )² U softmax(3 U U y:),
(A141)
VT V y;
softmax (3 vT n;)" vTV softmax(3 V yy)
(A142)
High variance of UT Vy.- To compute the dot product U. Vy M + K stored patterns are required (M of the u, and K of the vj). In contrast, the dot products U UO and V'. require only M or respectively K stored patterns. Therefore, UT V,, has higher variance than both UT U,, and VTV, yy.
Covariance structure extracted by U
U.U Uy, and VT 2, y.
Vg
36

---

Under review as a conference paper at ICLR 2022
The Jacobian J of the softmax p softmax(3a) is
((Ba) @softmax(Sa) 3 (diag(p) pp"),
(A143)
da
which a symmetric, positive semi-definite matrix with one eigenvalue of zero for eigenvector ((3a) is diagonally dominant since p,,(1 p.)l Ej Ipopyl = Pt P;P = Ps Pt 0.
Next we give upper bounds on the norm of J.
Lemma A1. For a softmax = softmax (xx) with m = max; pi(1 - pi). the spectral norm of the Jacobian of the softmax is bounded:
|J]| < 2 8
(A144)
||]| & 2mm 8 ||J
(A145)
< 2 m 5
(A146)
In particular everywhere holds
1
JJ]| < 3.
(A147)
Puax max; P4
0.5, then for the spectral norm of the Jacobian holds
|| < 2 2 2cB.
(A148)
Proof. We consider the maximum absolute column sum norm
All:
max
D aig
(A 1494
and the maximum absolute row sum norm
|
max
Σ lais
(A150)
We have for A = = 8 (diag(p) - -pp")
Σ aiyl
pi(1 p.) + Σ p.P
Bpp (1 2ps + ΣpD)
(A151)
2 3 Ps (1 -p:)) < 9
Σ layl
pj (1 p.) + Σ PsPs
BBj (1 - 2p, + Σ p.)
(A152)
i
2 (1 p,) < 2 m 3.
Therefore, we have
||]|| < 2 m 8
(A153)
| < B
(A154)
||]| < || & 2 3.
(A155)
The last incquality is a direct consequence of Hölder's inequality.
For 0 P: 1, have pr(1 -p+) & 0.25. Therefore, m 0.25 for all values of pt-
Pmax 2 1 - 2 0.5 (e < 0.5), then 1 - Puax < c and for p: * Pmax P: & c. The derivative dz(1 - /0x = 2r > 0 for 0.5, therefore æ=1 x) increases with x for == 1 - Pmax and for p; # Pmax I = Pi, we obtain p. pt) < e(1 - <) for all i. Consequently, have <(1 c).
0.5. Using
37

---

Under review as a conference paper at ICLR 2022
For the softmax p softmax(Ba) with Jacobian 0J/Oa J(Ba) (diag(p) pp") and for arbitrary N N-dimensional vectors b and c, we have
|Enety ps di
Σ I Gi x)
b" (33)) c = b² (diag(p) - pP") c = B Σ Endaa
(A156)
Therefore, 6" J(3a)e is 3 times the covariance between b and if component is drawn with probability of the multinomial distribution In our case the component is sample
Using the mean = 1/M E u4 the empirical covariance of data is
Cov(U) L/MUU á aû"
(A157)
M
M
[Cov(U)) Σ
1/M uss un
Σ
1/M
uik
Σ
1/M ual
(A158)
The weighted covariance (samples u, are drawn according to p;)
Cov(U) = U J(B a) U².
(A159)
[Cov(U)) 3 Pr Usk Un
Σ
Σ Ps Wik
Σ
Pt Uht
(A160)
which replaces 1/M from equal sampling by the pt, that is, u, sampled with probability Pt. The next theorem states how to express the dot product UT Uw. by weighted covariances of the data U.
Theorem A3 (Weighted Covariances). Using the weighted covariances
Cov(U, y1) = U 33 U y.) Cov(U, 21) = U x: U
(A161)
(B a) (' d()
(A162)
where the mean Jacobian J" is symmetric, diagonally dominant, and positive semi definite with spectral norm bounded by ||]= & 0.53.
The dot product UT. Uy. can be expressed by the weighted covariances
U Up. (ū Cov(U,,;;) x3)² (ū + -CCCCU y.)
(A163)
where the mean 1/MU1.
Proof. We apply the mean value theorem to the softmax with the symmetric, diagonally dominant, positive semi-definite Jacobian matrix Jm = fo J(Aa + (1 A)a') dA:
softmax(a) softmax(a') Jm (a a')
(A164)
We set a' 0 and use Sa instead of a, which gives:
softmax(B a) = 1/M1 + a, (3) = f J()
(A165)
which is exact. We obtain
softmax(3 UTx;) xx) 1/M + (T) UT
(A166)
softmax UT y:) = 1/M 1 + Jm yT UT
(A167)
The spectral norm of Jm is bounded by J < 0.59, since this bound holds for every J(A3a) in [m a) f/ J(ABa) dA according to Lemma A1.
38

---

Under review as a conference paper at ICLR 2022
The dot product between the anchor retrieval and the positive sample is:
U 2. Uv softmax(3 U² x.)" UºU softmax(3 U v.) (1/M1 JT (; a. U (1/M1 + 9,
(A168)
-1/MUU U (3 U (3
o') (1/M U (B
= (u + (Ov(U,2,) 22)² (u + Cov(U, yy)
where we used the mean ú = 1/MU1 and the weighted covariances
Cov(U,y)) U 1r U v.) U², Cov(U, = U J= U²
(A169)
The Jacobian Jm is symmetric, diagonally dominant, and positive semi-definite. The weighted covariance Cov(U, (U..) is the covariance if the stored pattern u, drawn according an averaged P: given by (.). Analog for weighted covariance Cov (V,.). When maximizing the dot product U2, Ug the normalized vectors x; and y, are encouraged agree on drawing the patterns u, with the same probability to generate similar weighted covariance matrices Cov(U..). ). If subsets of U have a strong covariance structure. then it can be exploited produce large weighted covariances and, in turn, large dot products of UTT Uy.- Furthermore, for a large dot product UT Uy æ; and y, have to be similar to one another to extract the same direction from the covariance matrices. All considerations are analog for V V
A.2 MUTUAL INFORMATION ESTIMATION
We follow the toy experiment discussed in Poole et al. (2019), Belghazi et al. (2018) and Cheng al. (2020) and experimentally confirm the superior quality of InfoLOOB for mutual information than InfoNCE. The dataset consists of samples (z,, 3/) drawn jointly from multivariate Gaussian distribution with correlation P where the dimension of the samples and y is set to d 20. We examine the performance of InfoLoob with and without Hopfield and InfoNCE at estimating mutual information of these samples. Due to the Gaussian distribution, the true value of mutual information can be calculated as T1x, y) = d log(1 1- We set the mutual information true value to the values (2.0, 4.0, 6.0, 8.0, 10.0, 14.0) by varying the value of p. At each MI true value, we sample data batches 1024 times, with batch size equal to 64, for the training of variational MI estimators. Figure shows that modern Hopfield networks reduce the variance of the model. For models trained on data with mutual information of 10 we observe average variance of approx. 0.67 for model without Hopfield and an average variance of approx. 0.33 for a model with Hopfield. For models trained on data with mutual information of 14 we observe an average variance of approx. 1.00 for model without Hopfield and an average variance of approx. 0.48 for a model with Hopfield.
In Figure Al we show the performance of method InfoLOOB with and without Hopfield estimating mutual information as well as InfoNCE. As expected estimates of InfoNCE have estimates that saturate at log(batch size). InfoLOOB without Hopfield exhibits good estimates of high mutual information while InfoLOOB with Hopfield accomplishes both good estimates of high mutual information with a decreased variance.
A.3 EXPERIMENTS
A.3.1 ABLATION STUDIES
As mentioned in the main paper, CLOOB has two new main components compared to CLIP: (1) the InfoLOOB objective instead of the InfoNCE objective and (2) the modern Hopfield networks. To assess which of the new main components CLOOB have led to the performance increase over CLIP, we performed ablation studies on the CC dataset. The results are reported in Table A1. First, we enhanced CLIP by replacing the InfoNCE objective with InfoLOOB (see column CLIP InfoLOOB). Next, we added modern Hopfield networks to the CLIP architecture and used retrieved embeddings instead of the original embeddings, while keeping the InfoNCE objective (see column Hopfield InfoNCE). Finally, we add modern Hopfield networks to CL.P and replace the InfoNCE objective
39

---

Under review as a conference paper at ICLR 2022
infoNCE
infoL008 without Hopfield
infoL008 with Hopfield
14
10
10
Mitial inferntion
matemnaioon
infermtion
+
True MI
(o((4)
5tops
stops
stops
Figure AI: The estimated mutual information of the InfoNCE objective saturates the batch size induced bound. The InfoLOOB objective trained with the same batch size with samples from the same correlated Gaussian distributions following (Belghazi et al., 2018; Poole et al., 2019; Cheng et al., 2020) is not limited by that bound and better estimates higher mutual information but suffers from higher variance. This is remedied by incorporating the modern Hopfield network.
Table Al: Influence of loss functions and Hopfield retrieval. InfoLOOB increases the performance of CLIP most of the tasks. The InfoNCE loss is not suited for the Hopfield approach as it saturates leading to worse performance. Hopfield with out of datasets compared to both CLIP models.
strongly improves the performance in
CLIP
Hopfield
Dataset
InfoNCE
InfoLOOB
InfoNCE
InfoLOOB
Birdsnap
1.94
2.37
1.67
2.53
Country211
0.62
0.63
0.54
0.76
10owers102
13.04
13.03
11.53
14.24
GTSRB
7.28
4.39
5.76
5.86
UCF101
21.00
19.14
20.56
22.29
Stanford Cars
0.90
1.33
1.24
1.37
ImageNet
20.31
22.13
19.04
24.21
ImageNetV2
20.63
21.65
18.97
23.80
with InfoLOOB (see column Hopfield InfoLOOB), As shown in Table Al the InfoLOOB objective increases the performance of CLIP in the majority of the datasets. We attribute this increase to the fact that InfoLOOB suffers less than InfoNCE from the "explaining away" problem. However, InfoLOOB even more effective for higher mutual information, that is, a richer covariance structure. Hopfield networks amplify the covariance structure in their retrieved embeddings. Though, this amplified covariance structure is saturation effect is caused by a richer covariance structure through Hopfield networks, which in turn leads to higher similarity between anchor and positive. Therefore, we see a performance drop when combining modern Hopfield networks with InfoNCE. Concluding, modern Hopfield networks are perfect match for InfoLOOB as they yield higher mutual information. Therefore, CLOOB strongly improves the performance on out zero-shot transfer learning tasks compared to CLIP.
for InfoNCE, as the saturation effect is stronger. The stronger
For CLIP with InfoNCE, the hyperparameter - is learnable parameter. For the other experiments, we use a fixed -1 of 30. The value for - was determined via hyperparameter search (see Section A.3.2).
In contrast to CLIP, we use learning rate scheduler with restarts (Loshchilov & Hutter, 2017) to be more flexible regarding the number of total training epochs and enable training up plateau. To investigate the influence of the learning rate scheduler, we performed experiments with and without restarts. Table A2 shows the zero-shot performance for the different downstream tasks for CLIP and CLOOB respectively. For both CLIP and CLOOB, the performance at the majority the tasks either increases remains roughly the same with restarts.
40

---

Under review as a conference paper at ICLR 2022
Table A2: Influence of learning rate scheduler. For most of the tasks the performance either increases or remains roughly the same with restarts for both CLIP and CLOOB.
CLIP
CLOOB
Dataset
w/o restarts
w/ restarts
w/o restarts
w/ restarts
Birdsnap
2.10
1.94
2.64
2.53
Country211
0.71
0.62
0.63
0.76
Flowers
11.00
13.04
11.50
14.24
GTSRB
6.16
7.28
5.05
5,86
UCF101
19.05
21.00
21.97
22.29
Stanford Cars
1.29
0.90
1.22
1.37
ImageNet
20.19
20.31
23.29
24.21
ImageNet V2
20.53
20.63
22.97
23,80
Table A3: Datasets used for zero-shot and linear probing. the case of several train or test sets per dataset we report the total number of samples. It should be noted that at the time of this work some Birdsnap images were not accessible anymore.
Dataset
Classes
Train size
Test size
Evaluation metric
Birdsnap
500
38,411
1,855
accuracy
Country211
211
42,200
21,100
accuracy
Flowers
102
2,040
6,149
class dass-weighted
accuracy
GTSRB
43
26,640
12,630
accuracy
ImageNet
1,000
1,281,167
50,000
accuracy
ImageNet V2
1,000
1,281,167
30,000
accuracy
Stanford Cars
196
8,144
8,041
accuracy
UCF101
101
28,747
11,213
accuracy
A.3.2 HYPERPARAMETERS
The hyperparameter search was done on validation split of CC with about 15,000 samples. For the hyperparameter -- several values considered (14.3, 30, 50, 70), where 30 leads to the best results for both YFCC and CC. Analogously to CLIP, we use the Adam optimizer (Kingma et al., 2014) with decoupled weight decay regularization (Loshchilov & Hutter, 2019). The weight decay is only applied to weights that are not gains or biases. As proposed in OpenCLIP (Ilharco et al., 2021) weight decay was set to 0.1. Different choices weight decay (0.2 or 0.05), did lead a relevant performance change. We use the same learning rate of 1 10 for CC and 5 10 10-4 for YFCC as used OpenCLII. For the hyperparameter 3 we considered values in the range of 8 resulted in the best performance for CC and 14.3 for YFCC. The batch size for CC was reduced to 512 due to computational restraints which did not result in performance losses. The batch size for YFCC was kept at 1024 reported by OpenCLIP since reduction resulted in significant drop in performance. The learning rate scheduler for all experiments is cosine annealing with warmup and hard restarts (Loshchilov & Hutter, 2017) with cycle length of epochs. For models trained YFCC the warmup was set to 10000 steps and for models trained on CC to 20000 steps.
to 20. value
A.3.3 DATASETS
For pretraining we consider two datasets, Conceptual Captions (CC) (Sharma et al., 2018) and YFCC100M (Thomee et al., 2016). The CC dataset consists of 2.9 million images and corresponding high-quality captions. Images and their corresponding notations for CC have been gathered via an automated process from the web and therefore represent a wide variety styles. Raw descriptions of images are collected from the alt-texi HTML attribute. Both images and texts are filtered such that only image-text pairs above a certain quality threshold are part of this dataset. The dataset we refer to YFCC is subset of the Yahoo Flickr Creative Commons 100 Million (YFCC100M) dataset. was created by filtering for images which contain natural language descriptions and/or titles in English resulting in 15 million image-caption pairs. The textual descriptions contain less
41

---

Under review as a conference paper at ICLR 2022
useful information than CC because they are not filtered by quality. Occasionally they also contain metadata like camera settings or web addresses.
We evaluate and compare our method several downstream classification tasks. We evaluate on the same set of datasets as CLIP reported for a model trained on YFCC. This set contains Birdsnap (Berg et al., 2014), Country211 1 (Radford et al., 2021), Flowers (Nilsback & Zisserman, 2008), GTSRB (Stallkamp et al., 2011), UCF101 (Soomro et al., 2012), Stanford Cars (Krause et al., 2013) and ImageNet (Deng et al., 2009). Additionally, we include ImageNet V2 in our analysis (Recht et al., 2019). Table A3 shows an overview of training and test set sizes, number of classes and the applied evaluation metric. In the case of several test sets per dataset the metric is calculated for every set individually and the average performance is reported. The set size in Table A3 corresponds to the total number of samples across all test and training sets of dataset respectively.
Birdsnap contains images of North American bird species, however our dataset is smaller than reported in CLIP as some samples are no longer available. The Country211 dataset was published in CLIP and is a small subset of the YFCC100m dataset. It consists of photos that can be assigned to 211 countries via GPS coordinates. For each country 200 photos are sampled for the training set and 100 for testing. For the Flowers102 images of 102 flower categories commonly occuring in the United Kingdom were collected. Several classes are very similar and there is a large variation in scale, pose and lighting. The German Traffic Sign Recognition Benchmark (GTSRB) was challenge held at the IJCNN 2011. The dataset contains images of german traffic signs from more than 40 classes. Note that two versions of this dataset exist, one used for the challenge and official dataset released after the competition. For CLIP the linear probing classifiers were trained using the competition training set but tested on the official test set. Stanford Cars contains images of 196 car models at the level of make, model and year (e.g. Tesla Model S Sedan 2012). UCF101 (Soomro et al., 2012) is a video dataset with short clips for action recognition consisting of three training sets and three test sets, We follow the procedure reported in CLIP and extract the middle frame of every video to assemble the dataset. The ImageNett Large Scale Visual Recognition Challenge was held from 2012 through 2017 and is one of the most widely used benchmarks for object detection and localization. Several years later ImageNet V2 assembled three new test sets with images from the same 1,000 classes to test for generalization of models optimized for the original ImageNet benchmark. Every test set comprises 10,000 samples.
A.3.4 ZERO-SHOT EVALUATION
Class names for all downstream tasks were adopted from CLIP, that is, among other changes special characters like hyphens or apostrophes were removed. Furthermore, some class names of the datasets were slightly changed (e.g. "kite" to "kite (bird of prey) in ImageNet). For zero-shot evaluation. we use the same prompt templates as published in CLIP. Depending on the dataset the number of prompts can vary from one prompt (e.g. "a photo of (label) }, a type of bird." " for Birdsnap) up to 80 prompts for ImageNet covering various settings (e.g. "a cropped photo of (label) ", "a origami (label), .." ."). In case of several prompts an average embedding over all prompt embeddings is calculated. Figure A2 shows the zero-shot results for all evaluation tasks with the ResNet-50x4 model reported in Table 4.
A.3.5 LINEAR PROBING
We try to follow the evaluation procedure in Radford et al. (2021) as closely as possible. We note one difference with respect to the implementation: Instead of scikit-learn's 's logistic regression using the L-BFGS solver, we use cuML's logistic regression classifier with L-BFGS algorithm to utilize GPUs for efficiency. All hyperparameters are the same as described in Radford et al. (2021), the maximum number of iterations was set to 1000, and the L2 regularization strength À was determined by using a parametric binary search.
We tried to reproduce the CLIP results with the correspondingly published models, however, failed to produce the exact numbers. This could be due to several factors:
The train and validation split. Same in Radford et al. (2021). the provided validation set to perform the hyperparameter search. When there is none provided, we use a random half of the training dataset for validation.
42

---

Under review as a conference paper at ICLR 2022
Table A4: Linear probing results for the reimplementation of CLIP and CLOOB using different
ResNet architectures trained on YFCC. The performance of CLOOB scales with increased encoder size
CLIP RN-50
CLOOB RN-50
CLOOB RN-101
CLOOB
Dataset
RN-50x4
Birdsnap
50.9
56.2
58.1
62.2
Country211
19.5
20.6
21.8
24.2
Flowers 102
94.8
96.1
96.1
96.2
GTSRB
82.5
78.9
77.9
80.6
UCF101
75.2
72.3
72.8
75.3
Stanford Cars
36.2
37.7
39.0
44.3
ImageNet
66.9
65.7
67.0
69.7
ImageNet V2
60.2
58.7
60.3
62.2
In case of a tie the validation score, we use the maximal X for the strongest regularization. We note though that came closer reproducing the results published in CLIP when using the mean À over all ties when these exist.
For the Birdsnap dataset, the resources that we have got online at the time of this writing could be different from the resources that CLIP's authors obtained the time.
Linear probing evaluation of YFCC-pretrainee models is shown in Table A4. Comparing reimplementation of CLIP and CLOOB with ResNet-50 encoders, we observe mixed results. The reason for this effect might be attributed the observed lask-dependence of multimodal models (Devillers et al., 2021). Another potential reason is that the benefit of the restrictions to more reliable patterns that occur in both modalities does directly translate an evaluation just the encoding part of one modality. Again, as expected in self-supervised training, increasing the capacity of the CLOOB models benefits accuracy.
A.4 REVIEW OF MODERN HOPFIELD NETWORKS
We briefly review continuous modern Hopfield networks that used for deep learning architectures. They are continuous and differentiable, therefore they work with gradient descent in deep architec- tures. They retrieve with one update only, therefore they can be activated like other deep learning layers. They have exponential storage capacity, therefore they can tackle large problems. Hopfield networks are energy-band -haeed binary associative memories, which popularized artificial neural networks in the 1980s (Hopfield, 1982; 1984). Associative memory networks have been designed to store and retrieve samples. Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Chcol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott Arian, 1987; Horn Usher, 1988; Caputo Niemann, 2002; Krotov Hopfield, 2016). In contrast to these binary memory networks, very high storage capacity. These modern Hopfield networks for deep learning architectures have an energy function with continuous states and can retrieve samples with only update (Ramsauer et al., 2021; 2020). Modern Hopfield Networks have been successfully applied to immune repertoire classification (Widrich al., 2020) and chemical reaction prediction (Seidl et al., 2021).
continuous associative memory networks with
We assume a set of patterns { uN C Rd that are stacked as columns to the matrix U (ui...., uN) and a state pattern (query) dRd that represents the current state. The largest norm of stored pattern is M = max; |us|| Continuous modern Hopfield networks with state & have the energy
N
=
log
Σ
exp(Bu
+ 3oo1oog + ETE + 2
M
(A170)
2
For energy E and state &, the update rule
gnow
f(E: U p U softmax (BUT
(A171)
43

---

Under review as a conference paper at ICLR 2022
COPNCIIMA 11033
CONMETAAr 1/1300
27100
Oryara HMOS
Gitle
CONNCIIRE 51033
Freeingg radine
comerarc 15/000
coreErranc T11000
L580
Gene Ska Haron
1,300
5.30
wajed
CoreCcioN 1911
G
camerane 1/11
Spaln
COnVE 26011
vnlecci with
and vol
red reth Sppenshingg
comeerare 3611
11/13
Iastaacinn marning
Desgs Caller Woger P0r3
Drysar Appen SUN
camerar
Chease Toxe
comeec
Morue 200r
Tonalo
Eadelaal
Figure A2: Visualization of zero-shot classification of three examples from each dataset. The follow- ing datasets are used (top to bottom): ImageNet, ImageNet V2, Birdsnap, Country211. Flowers I 02, GTSRB, Stanford Cars and UCF101, The ground truth label is displayed above the picture. The bar plots show the softmax values of the top 5 classes.
has been proven to converge globally to stationary points of the energy E, which are almost always local minima (Ramsauer et al., 2021). The update rule Eq. (A171) also the formula of the well- known transformer attention mechanism (Ramsauer et al., 2021), therefore Hopfield retrieval and transformer attention coincide.
44

---

Under review as a conference paper at ICLR 2022
The separation A, of pattern u, is defined as its minimal dot product difference any of the other patterns: A = minj,jss (u, u, u u,). A pattern is well-separated from the data 2 + 3 log (2(N - 1)N (M2). If the patterns u, are well separated, the iterate Eq. (A171) A 2 3N
converges to fixed point close to stored pattern. If some patterns are similar to one another and, therefore, not well separated, the update rule Eq. (A171) converges of the similar patterns. This fixed point is metastable state of the energy function and averages over similar patterns.
fixed point close to the mean
The next theorem states that the update rule Eq. (A171) typically converges after one update if the patterns are well separated. Furthermore, states that the retrieval error exponentially small the separation A..
Theorem A4 (Modem Hopfield Networks: Retrieval with One Update). With query &, after one update the distance of the new point ((E) to the fixed point ux is exponentially small in the separation As The precise bounds using the Jacobian J = 8f(E) and its value J in the mean value theorem DE
are:
(I) wlll & 1 112 l1E u | & 2BN M 11- exp(- 3 (Ai - max(le uill luu uull} M))
(A172)
(A173)
For given e and sufficient large Di. we have (lf(EE) uil || < e, that is, retrieval with one update. The retrieval error (() uull of pattern u, bounded by
lIf(E) - -uil & (N - exp(- B (A. - 2 maxflle - uill. 1|,, - -u.11 M)) M
(A174)
For proof see (Ramsauer al., 2021).
The main requirement of modern Hopfield networks to be suited for contrastive learning is that they can store and retrieve enough embeddings the batch size is large. We want to store a potentially large set of embeddings. We first define what we mean by storing and retrieving patterns from modern Hopfield network.
Definition A1 (Pattern Stored and Retrieved). We assume that around every pattern u, a sphere S; is given. We say u, is stored if there is a single fixed point un E Sq to which all points & S; converge, and S, nS, 0 for + j. We say u; is retrieved for a given e if iteration (update rule) Eq. (A171) gives a point ã; that is at least e-close the single fixed point u, S,. The retrieval error is Ø uzll-
As with classical Hopfield networks, we consider patterns on the sphere, i.e. patterns with fixed norm. For randomly chosen patterns, the number of patterns that can be stored exponential in the dimension d of the space of the patterns (u; E (R.).
Theorem A5 (Modem Hopfield Networks: Exponential Storage Capacity). We assume a failur probability 0 < p & 1 and randomly chosen patterns the sphere with radius M == K KVd 1. We define == (1 + In(2.333 p(d - 1))), b - and c = Wo(exp(o+ where Wo is
2K²
the upper branch of the Lambert W function (Olver et al., 2010, (4.13)), and ensure 2 Then with probability 1 the number of random patterns that can be stored is
VP
N 2 vp +
(A175)
Therefore is proven for e 2 3.1546 with 3 1, K = 20 and = 0.001 (a + In(b) 1.27)
and proven for e 2 1.3718 with 3 = 1, K = 1, d 75, and p 0.001 (a + In(b) < -0.94).
For proof see (Ramsauer al., 2021).
This theorem justifies to use continuous modern Hopfield networks for using retrieved embeddings instead of the original embeddings for large batch sizes. Even for hundreds of thousands of embed- dings, the continuous modern Hopfield network able to retrieve the embeddings the dimension of the embeddings is large enough.
45

---

Under review as a conference paper at ICLR 2022
A.5 FURTHER RELATED WORK
Multiple works have proposed improvements to InfoNCE. Joint Contrastive Learning (JCL) studies the effect of sampling multiple positives for each anchor. (Cai al., 2020). Sampling negatives around each positive leads to higher bias but lower variance than InfoNCE (Wu et al., 2021). InfoNCE has been generalized C-InfoNCE and WeaC-InfoNCE, which are conditional contrastive learning approaches to remove undesirable information in self-supervised representations (Tsai et al., 2021). ProtoNCE is a generalized version of the InfoNCE, which pushes representations to closer to their assigned prototypes (Li et al., 2021). ProtoNCE combines contrastive learning with clustering. SimCSE employs InfoNCE for contrastive learning to learn sentence embeddings (Gao et al., 2021). InfoNCE has been extended to video representation learning (Han et al., 2020).
Many follow up works have been based on the CLIP model. The CLIP model is used in Vision-and- Language tasks (Shen al., 2021). The CLIP model guided generative models via additional training objective (Bau et al., 2021; Galatolo ct al., 2021; Frans ct al., 2021) and improved clustering of latent representations (Pakhomov et al., 2021). I). It is used in studies of out of distribution performance (Devillers et al., 2021; Milbich et al., 2021; Miller et al., 2021), of fine-tuning robustness (Wortsman et al., 2021), of zero-shot prompts (Zhou et al., 2021) and of adversarial attacks to uncurated datasets (Carlini & Terzis, 2021). It stirred discussions about more holistic evaluation schemes in computer vision (Agarwal al., 2021). Multiple methods utilize the CLIP model in straightforward way to perform text-to- text-to-video retrieval (Fang et al., 2021; Luo et al., 2021; Narasimhan et al., 2021).
46


Reviews:
[{"review_id": "SLj_0CXosPZ", "reviewer": "Reviewer_jmHN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This paper is well written and easy to follow. There is almost no typo in the paper.\n\n2. The proposed method is interesting. The introduction of Hopfield networks into contrastive learning to reduce the high variance of InfoLOOB is well motivated. In particular, CLOOB is easy to be implemented in practical applications.\n\n3. The experiments on zero-shot datasets are well extensive and truly demonstrate the advantages of the proposed CLOOB when compared with the latest method CLIP. I appreciate the effectiveness of the proposed method in real-world applications.\n\n4. The content of the appendix is substantial. The proof seems sound and correct, after I checked it.\n\nWeaknesses:\n\n1. The novelty of the proposed method seems not big enough. In my opinion, the proposed method is just to add modern Hopfield networks on top of the encoder to transform the original embedding to an informative one.  The total objective of the loss function in Eq. 13 is still the common-used  InfoLOOB loss. From this perspective, it seems like that the proposed method is just a feature-engineering algorithm, whose novelty, in my opinion, is not convincing enough. However, it is still glad to see that such simple algorithm is efficient in practical applications.\n\n2. My major concern is the originality of the theoretical results of this paper. As far as I can tell, both Theorem 1 and Theorem 2 are the results, or can be directly obtained, from previous work. For example, the InfoNCE as the lower bound of mutual information (MI) is well-known result in the work of Poole2019ICML (cf Eq. 12 in “On Variational Bounds of Mutual Information” ), the InfoLOOB as the upper bound of MI can also be directly derived from Eq. 13 in Poole2019ICML. Therefore, in my opinion, there is no need to present the whole detailed proof of these theoretical results in Appendix, as such proof technique is not originated from the authors. Rigorously speaking, Theorem 1 and Theorem 2 in the main paper need citations as they are just the theoretical results from previous works. Authors of CLOOB could not use such theoretical results as their own.\n\n3.  In view of Weakness 2, the theoretical contribution of this paper is limited. In my opinion, authors should give more theoretical analysis of their proposed CLOOB method, such as whether the introduction of Hopfield networks can theoretically reduce the variance of the learned model, instead of presenting the well-known lower and upper bounds on MI as their main theoretical results. ", "summary_of_the_paper": "This paper proposed a new contrastive learning method called CLOOB, which minimized the leave-one-out upper bound (InfoLOOB) on mutual information with the modern Hopfield networks. Concretely, Hopfield networks replace the original embeddings by retrieved embeddings in the InfoLOOB objective.  The retrieved embeddings are more robust as they capture the common covariance structure of all sample embeddings, leading to a stable performance of InfoLOOB. Extensive experiments on several zero-shot datasets show that the proposed CLOOB method outperforms the well-known CLIP model across all considered architectures.", "summary_of_the_review": "I appreciate the effectiveness of the proposed method. However, considering the limited theoretical contribution of this work, and insufficient novelty of the proposed method, I could only give a borderline score (incline to rejection) for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IUrS2cAZWhw", "reviewer": "Reviewer_z4BR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\n- The method is clear and the paper well written and structured.\n- The proposed CLOOB approach outperforms the chosen baseline on most tasks, demonstrating a clear performance gain.\n- Ablation study A1 does seem to demonstrate that the objective introduced has strong synergies with the Hopfield network approach. This justifies the usefulness of the different components.\n\nConcerns:\n- The ablation study seems to indicate that InfoLOOB does not work well using the CLIP framework. Could the authors expand on why they thing this is the case?\n- While it is true that the authors design a learning algorithm that functions well against their chosen baseline, overall most of the components (the InfoLOOB, modern Hopfield networks, ...) are not intrinsically novel. This does not detract from the performance gains presented in the paper however.", "summary_of_the_paper": "The paper introduces an unsupervised learning technique called: \"Contrastive leave-one out boost\", which relies on a contrastive learning objective called \"InfoLOOB\". The authors compare this objective in detail with the classical InfoNCE objective, and compare the performance of their method with CLIP on a set of zero-shot transfer learning tasks (pre-training occurs on two datasets, conceptual captions and YFCC).", "summary_of_the_review": "The paper is well written and interesting. The authors combine different components to design CLOOB, a learning algorithm that shows performance gains compared to the chosen baseline. Despite this, the components used in the approach are not novel, they are the result of previous work, meaning that the technical contributions mainly consist in finding ways to adequately combine Hopfield networks and the InfoLOOB objective. I would still recommend acceptance as I do not feel lack of technical novelty alone should detract from the empirical merits of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4IucyYqtew", "reviewer": "Reviewer_BgtC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n - The paper is well written, with clear motivation, careful algorithm derivation, and detailed experiments + ablations. The comparison between InfoLOOB and InfoNCE losses (Section 2) are carefully made and technical sound (myself didn't rigorously go through the proof, though)\n - Good empirical improvements over CLIP.\n\n**Weaknesses**\n - Contribution / novelty is small: neither InfoLOOB nor Modern Hopfield Network is new\n - In addition to image-text contrastive learning, would love to see how it applies to other contrastive learning tasks (e.g. image-image)", "summary_of_the_paper": "This paper presents a new contrastive learning algorithm that comes with two improvements: a) replacing InfoNCE loss with InfoLOOB loss, b) replacing normal deep neural network with Modern Hopfield Network. When applied to image<->text contrastive learning, the new algorithm shows good improvements over baseline (CLIP) on a number of zero-shot image classification benchmarks.", "summary_of_the_review": "While empirical results are good, the overall contribution of the paper seems fall below the bar. It simply applies a combination of two existing techniques to image-text contrastive learning, and doesn't show how it broadly applies to other contrastive learning tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-9V7LlEPEdu", "reviewer": "Reviewer_vr8w", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1) Writing: The paper is generally carefully written, and easy to follow and understand. The reviewer enjoys reading the paper, though did not check the theorems in detail.   \n\n2) Novelty: The proposed CLOOB method is novel. It combines the InfoLOOB objective with modern Hopfield network for the training of a CLIP-like model. This has not been investigated before. Since CLIP is a hot topic in the field, it will be interesting for the community to know how CLIP can be further enhanced. \n\n3) Experiments: The authors have conducted comprehensive experiments on zero-shot transfer learning, in order to demonstrate the effectiveness of CLOOB when compared with CLIP.  \n\nWeaknesses: \n\n1) Method:\n\na) Typically, people will try to maximize an MI lower bound, and minimize an MI upper bound. In this paper, the authors try to maximize an MI upper bound. More discussion and rationales behind this design will be appreciated. \n\nb) Since the proposed method is general, besides CLIP, did the authors try to apply it to self-supervised visual representation learning, such as MoCo and SimCLR etc.\n\nc) The authors use InfoLOOB as the objective. Since InfoLOOB is an upper bound, a natural question is that why not also try using CLUB (Cheng et al., ICML 2020) as the objective?\n\nd) What exactly is a modern Hopfield network? Eqn. (9) and (10) tried to explain this, but they are not clear enough to me. Why they are better? What's the original application they are proposed for? Since this is a core part of the proposed method, a more detailed description is needed. \n\n2) Experiments:\n\na) From results in Figure 1, it is not easy to see clearly variance of InfoLOOB is reduced via modern Hopfield network. Is there any quantitative measure on the variance?\n\nb) It would be interesting to also test ViT backbones, instead of ResNet backbones. \n\nc) The ablation study in Sec. 4.1 seems important, maybe the authors should consider move them back to the main text if space permits. \n\nd) In Table 3, what are the performance of CLIP under stronger RN-101 and RN-50x4 backbones? It will be interesting to see how the performance gap changes while making the visual backbone stronger.\n\n\nTypo:\n\nFigure 2, caption, 4-th and 5-th row, the \"image embedding U_{y_i}\" should be \"text embedding\"?", "summary_of_the_paper": "By using the InfoNCE loss for model training, CLIP has achieved great success. In this paper, the authors propose CLOOB, short for \"Contrastive Leave One Out Boost\", where modern Hopfield networks are used together with the InfoLOOB objective. InfoLOOB is a leave-one-out upper bound of mutual information, and modern Hopfield networks replace the original embeddings by retrieved embeddings. Results show that CLOOB outperforms CLIP at zero-shot transfer learning across multiple architectures and datasets. ", "summary_of_the_review": "In summary, this paper proposes a novel method for CLIP-like model training, and results demonstrate the effectiveness of the proposed method on zero-shot transfer learning. However, the reviewer also thinks that this paper can be made stronger (see details in my comments), therefore, the reviewer decides to give a Borderline Accept recommendation at this moment.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four experts reviewed the paper and provided mixed recommendations. All reviewers found the experimental results strong, but they have different views about the technical novelty. Three reviewers considered the technical novelty as a weakness of the paper, but Reviewer z4BR was less concerned about it than the other two. After AC carefully read the paper and the authors' responses, AC agreed with the reviewers that the combination of InfoLOOB and modern Hopfield networks, which were both existing works, is incremental despite the empirical results. Besides, AC agreed with Reviewer jmHN that the theoretical results are not significant enough and could be moved to Appendices. While the empirical results are strong, they could not answer how the trend would change with bigger models and bigger datasets. Hence, while the paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject