Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric
Authors: Toshimitsu Uesaka
Taiji Suzuki
Yuhta Takida
Chieh-Hsin Lai
Naoki Murata
Yuki Mitsufuji
Venue: ICLR 2025 Spotlight (2025)
Keywords: contrastive learning
representation learning
multimodal representation learning
theoretical analysis
InfoNCE
pointwise mutual information
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a new multimodal representation learning method and theoretically show benefits of our method.

Abstract: In typical multimodal contrastive learning, such as CLIP, encoders produce one
point in the latent representation space for each input. However, one-point representation
has difficulty in capturing the relationship and the similarity structure of a
huge amount of instances in the real world. For richer classes of the similarity, we
propose the use of weighted point sets, namely, sets of pairs of weight and vector,
as representations of instances. In this work, we theoretically show the benefit
of our proposed method through a new understanding of the contrastive loss of
CLIP, which we call symmetric InfoNCE. We clarify that the optimal similarity
that minimizes symmetric InfoNCE is the pointwise mutual information, and show
an upper bound of excess risk on downstream classification tasks of representations
that achieve the optimal similarity. In addition, we show that our proposed
similarity based on weighted point sets consistently achieves the optimal similarity.
To verify the effectiveness of our proposed method, we demonstrate pretraining of
text-image representation models and classification tasks on common benchmarks.

Published as a conference paper at ICLR 2025
WEIGHTED POINT SET EMBEDDING FOR MULTIMODAL CONTRASTIVE LEARNING TOWARD OPTIMAL SIMILAR- ITY METRIC
Toshimitsu Uesaka", Taiji Suzuki2.3, Yuhta Takida', Chich-Hsin Lai', Naoki Murata', Yuki Mitsufujii
'Sony AI, The University of Tokyo, "RIKEN AIP, *Sony Group Corporation
toshimitsu wwsakal sony. com, taijiemist. i u-tooko. (p,
takida, whwch-hsichhcccc.e naoii .murata,
ABSTRACT
In typical multimodal contrastive learning, such CLIP, encoders produce one point in the latent representation space for each input. However, one-point repre- sentation has difficulty in capturing the relationship and the similarity structure of a huge amount of instances in the real world. For richer classes of the similarity, we propose the use of weighted point sets, namely, sets of pairs of weight and vector, as representations of instances. In this work, we theoretically show the benefit of our proposed method through new understanding of the contrastive loss of CLIP, which we call symmetric InfoNCE. We clarify that the optimal similarity that minimizes symmetric InfoNCE is the pointwise mutual information, and show an upper bound of excess risk on downstream classification tasks of representa- tions that achieve the optimal similarity. In addition, we show that our proposed similarity based on weighted point sets consistently achieves the optimal similarity. To verify the effectiveness of our proposed method, we demonstrate pretraining text-image representation models and classification tasks on common benchmarks.
INTRODUCTION
CLIP (Radford et al. 2021 and ALIGN Jia et al. 2021) established one of the most common frameworks for multimodal representation learning (Guo al. 2019). this framework, to obtain
the text-image representation, two encoders that map inputs from different modalities onto shared space are trained with a contrastive loss (Chopra et al. 2005). Recent studies have shown that CLIP model pretrained
large-scale text-image dataset provides transferable features to various downstream tasks such as linear classification (Radford et al. 2021 Jia et al. (021). text-to-video retrieval (Lin et al. 2022), and text-conditioned image generation (Ramesh al. 2022). Other
work has shown that a CLIP model can be used to feed vision information to large language models
(Alayrac al., 2022). In addition text and image modalities, this multimodal contrastive learning framework can be applied to other combinations of modalities such as texl-audio representations
(Elizalde et al. 2023) and combinations of more than modalities (Guzhov et al. 2022
Wu et al.
2022 Girdhar et al. 2023).
Despite the success of CLIP models, still arguable whether the similarity structure and repre- sentations they provide are suitable for modeling concepts in the real world. Typical CLIP encoders transform each input image or text into one point embedding in a latent space, and encoders are trained to enhance the similarity of paired concepts in training dataset, which is defined by the cosine similarity of their embeddings. However, concepts in the real world have a broadness that raises the relationship of inclusion and many-to-many correspondences. For example, the text "a photo of dogs" can conceivably be the caption of any number of different images, while another text, "a photo of poodles", could be the caption of the subset of dog photos, and the photo of poodles should be linked to the multiple captions. Considering these relationships, representations of concepts should be provided in manner that goes beyond singular point and exhibit innate broadness.

---

Published as a conference paper at ICLR 2025
In this paper, we propose the use of a weighted point set, namely set of pairs of scalar weight and a vector point, as the representation of each concept. which we call Weighted Point Set Embedding (WPSE). We define the similarity of two weighted point sets with kernel function that defines the similarity of two points. We also provide a theoretical rationale of the proposed weighted point set embedding through new understanding of the contrastive loss utilized in CLIP, which we call the symmetric InfoNCE loss. First, we highlight the fact that minimization of the symmetric InfoNCE loss is achieved when the similarity of two features in the loss represented by the pointwise mutual information. Second, we show, under some assumptions, that the optimal (possibly nonlinear) classifier in downstream classification tasks can be constructed by linear classifier over learned representations when the optimal similarity is achieved. Last, we show that the proposed similarity of weighted point sets has richer representation capacity than the cosine similarity, which is the bilinear similarity in the latent space. Moreover, to demonstrate the effectiveness of the proposed method, we conduct experiments on the Conceptual Caption datasets and common benchmark datasets.
2 RELATED WORK
2.1 MULTIMODAL CONTRASTIVE REPRESENTATION LEARNING IN PRACTICE
CLIP (Radford et al. 2021 and ALIGN JJa et al. 2021 utilize contrastive loss to obtain text-image
representations, inspired by a series of studies of deep metric learning and unimodal contrastive learning such as multi-class N-pair loss (Sohn 2016), InfoNCE (Oord et al., 2018), SimCLR (Chen
jtt al. 2020), and VIRT (Zhang et al.) 2022). ). Both works have shown the success of pretrained models with large-scale paired datasets and the contrastive loss, which we call the symmetric InfoNCE in this paper, in zero-shot settings and downstream tasks.
One approach to extending this contrastive framework is to modify the similarity in the symmetric InfoNCE loss. Fürst et al. (2022) proposed using modern Hopfield networks for computing similarities to enrich the covariance structure of data, while also replacing the InfoNCE with the InfoLOOB. Desai et al. (203) proposed using the Lorentzian distance in a hyperbolic space as the similarity to capture hierarchy structure of visual and linguistic concepts. Following this approach, enriching the class of the similarity based on a nonlinear kernel and weighted point sets. In contrast to the above studies, we provide an analysis of excess risk in downstream linear classifications.
propose
2.2 THEORETICAL UNDERSTANDING OF CONTRASTIVE LOSS
Early works attributed the success of the InfoNCE loss (Oord et al., 2018) to the fact that it is a lower bound of mutual information and its optimization leads to maximization of the mutual information between two views of data (Hjelm al., 2019: Bachman et al.) 2019 TTian et al. 2020). However. Tschannen et al. (2020) demonstrated through thought experiment and empirical results that maximizing tighter bounds on mutual information can result in worse representations. Li et al. (2021) also showed that different representations with the same mutual information can exhibit different qualitics. In an alternative perspective. Wann & Isola (2020) investigated alignment and uniformity to understand the InfoNCE. This idea has affected subsequent works theoretical analysis of contrastive learning (Li al. 2021] Zimmermann et al. 2021 Huang et al. 2023).
Regarding the theoretical relationship to downstream tasks, (Saunshi et al. (2019) showed that the
downstream classification loss is upper bounded by quantity monotonically increasing with respect to
the contrastive loss. Although|Saunse et al. (2019) relied on the strong assumption of the conditional independence of two samples, subsequent studies have mitigated this problem. HaoChen et al.) (202T) proposed the spectral contrastive loss and provided an upper bound of the linear probe performance based on the augmentation graph. Tosh et al. (202T) analyzed the excess loss of linear predictors on the landmark embedding from the perspective of multi-vieu redundancy. Wang et al. (202)) showed upper and lower bounds for downstream performance through the conditional feature variance and the augmentation overlap effect. AAsh et al. (2022) investigated upper bounds of a supervised loss in terms of the number of negative samples. |Huang al. (2023) analyzed the performance of the nearest neighbor classifier through (a, -ugummmntttion, Shi et al. (2023) investigated the trade-off between label efficiency and universality under assumptions of linear data. Waida et al. (2023) proposed the kernel contrastive loss and showed an upper bound of the classification error. Chen et al. (2024)
studied zero-shot transfer capability of CLIP with awareness of unexpected positive pairs. Zhai
2

---

Published as a conference paper at ICLR 2025
al. (2024) analyzed self supervidd representation learning through the lens of RKHS induced by
augmentations.
However, we argue that there are still three issues to be resolved in terms of understanding the framework of CLIP. First, some works provided only upper bounds of downstream losses. If there certain gap between the upper bounds and the optimal value, reducing the contrastive loss does not always mean a better performance in the downstream task. Second, some works changed the target of theoretical analysis from the actual setting of CLIP or InfoNCE and provided guarantees on their proposed losses or different features from usual contrastive learning. Last, some upper or lower bounds included various statistics (e.g., variance) of the obtained presentations. While such bounds are useful when perfect alignment is achieved, the perfect aligmment is not always practical in the case of multimodal learning, where paired samples are not generated by augmentations of the same instance and a data sample in modality has relationship to many samples in another modality.
Our work differs from the above studies in the following ways. First, we consider not only an upper bound of the performance but also the gap from the optimal classifier. Second, we analyze the symmetric InfoNCE and linear classifiers that are constructed using an approach similar to the actual setting of CLIP. Last, our assumptions for theoretical results are relatively mild in the case of multimodal representation learning, which is explained in Section 4.2
3 PROBLEM SETUP
In this section, we introduce the notations and problem settings that we use in following sections. We formalize the multimodal contrastive representation learning and the downstream linear classification task, which is commonly utilized to evaluate representation learning methods (Chen et al. 2020 Radford et al. 2021).
3.1 CONTRASTIVE REPRESENTATION LEARNING AND SYMMETRIC INFONCE
Let X' and y denote the input space of two modalities. For the sake of simplicity, we focus on text-image representation learning, and we denote the image space by X' and the text space by y Let px,v(2: y) denote the density of the joint data distribution of random variables (X, Y) defined over X' V, and let px(x)) (1) and py (y) denote the density of the marginal distribution of X and Y. respectively. If there is no ambiguity, we omit subscripts of probability (density) functions such as p(x, v). p(z), and p(y). We denote the conditional probability density of y given as py (y x)). For a subset ) C y, we denote the probability with which Y E y" as PY")") = SyEV' Ply)dy. We also denote the conditional probability of subset Y" given x as Py (Q" | x) = SyeY ply x)dy. For probability density function p. we denote the support of the probability as supp
Given a batch of N image-tex: pairs (I1, (11,8))..... (ÆN, MN) ~ PX,Y, CLIP (Radford et al. 2021 introduced the following contrastive loss to train an image encoder fx: X + R text encoder fy: y + R2d and trainable temperature parameter T E R
exp(fx(e.) (o( fy(())/T)
f1xx, =
N
Ek (1(0)))
(xp(fx((3) T fu(Ui)/T)
N In E (xp([x(xx))) [T (vx))//
We call this the symmetric InfoNCE loss. By minimizing it, the similarity of two features from paired samples (i%, y1) is expected to be large, and the similarity of two features from independent samples z, and yj (i # is expected to be small. Here, the similarity of two features is measured by the cosine similarity fxxx) [y()). Note that the features fx(x)) and fy(v) of typical CLIP are L2-normalized. For a generalized formulation, we replace the scaled similarity fx(x)" fy(v)T with function 9: X x y -> R of two samples (x, y) X' x y. In addition, following the asymptotic form of the InfoNCE in Wang & Isola (2020), we consider the population expectation form of the symmetric InfoNCE. By considering the limit as N - oc, we have the population expectation form
3

---

Published as a conference paper at ICLR 2025
of the symmetric InfoNCE:
(g) 1 In
exp (r, v)
1
exp (x, y) Epy (v)) lexp g(z, 3/)
E m().2))
In
(2)
p(a.v)
(xx (2)) lexp gax,, , 8)]
where we omit the constant term that comes from In N.
3.2 DOWNSTREAM CLASSIFICATION TASK
As common evaluation of the learned representations with the symmetric InfoNCE, we consider supervised classification task with K labels. For an integer M. we define [M] := {1..... M). Let C denote a random variable for labels. Let Pa | x) be the conditional probability of the label e E [K] given the data E X. We define p(3,, c) (c x)px (1) as the density of the joint distribution of data x and its label c. We assume that pairs of data and its label (x, c) can be drawn from p(r, c). In this supervised learning setting, classifier h: X' - R is often trained by minimization of the where h(z); denotes
softmax cross-entropy loss given by Laup (h) == Ep(a.c) In exp)
Eis exp
the i-th entry of h(x:) E RKK In downstream linear classifications after the contrastive learning. h is constructed as a linear classifier over the learned representation. Given the encoder fxx we formalize W fx(x)) + b. where W IR is weight and b e R
this linear classifier as h(;; fx) a bias. With this h(x; fx), the downstream classification task formalized as the minimization problem of Sau with respect to W and d: m bERK Esup(h(x; fx)).
THEORETICAL GUARANTEE VIA POINTWISE MUTUAL INFORMATION
In this section, derive the upper bound for the performance of downstream classification tasks. First, we highlight that the optimal similarity of the symmetric InfoNCE loss is represented by the pointwise mutual information. Second, we show that if the optimal similarity is obtained, there is linear classifier on the learned representation that is close to the optimal (possibly nonlinear) classifier. Last, we investigate an error caused by the deviation from the optimal similarity.
4.1 POINTWISE MUTUAL INFORMATION AS OPTIMAL SIMILARITY
Our analysis starts with the following fact that the optimal similarity of the symmetric InfoNCE represented by the pointwise mutual information (Oord et al. 2018 Zhang al. (003).
Proposition 4.1 (Restatement of Proposition in Zhang et al. (202))) Let X and Y denote two random variables having the joint probability density p. Then, the mutual information of X and
Y. IXI(XYY) Ep(a.)) In p(u.u)
is an upper bound of -LNCE(9). Moreover the function g
p (x)p(s)
satisfies g(x, v) = In p(z)p(y)
p(z.))
+ const, then the equality 1(X,Y) = -LNCE(g) holds.
In other words, when we consider the minimization problem of LNCE in terms of the measurable function 9 over X' x y, the optimal similarity is equal to the pointwise mutual information up to constant, We denote this optimal similarity by g" (1x y) In p(a.y)
I for some I E R.
p(=)p(()
4.2 POINTWISE MUTUAL INFORMATION ESTIMATOR LEADS TO GOOD LINEAR CLASSIFIER
Next, we show that, under some conditions, there exists linear classifier over learned representations that close the optimal classifier h- arg min; (aup(h) if we successfully obtain encoders that achieve the optimal similarity g (xr y). It is known that the log probability of the label c conditioned by data is the minimizer of Lxup up to a constant: h" (xxx); = Po x)) const, for i [ This is because Loup is represented by using the cross entropy H :) as follows: Loup (h) exp ((r)e
Ep(w) (P x), Qc(C' xx; h))]. where Qc(c) x; h) = Σx, exp M(x),
for E [K].
To explain our theoretical results, we define several probability (density) functions. We consider K disjoint subsets V, (i [K]) C y, i.e., for + j. ). ny = 0. Let 5 = Vi U32 U... U.K. Note that y is not necessarily equal to y. We assume that P().) A 0 for every We define the conditional
4

---

Published as a conference paper at ICLR 2025
p():) if y Vr, otherwise 0. Note that py pY (4a) is probability density function on y (i.e., JnEy Py (y/ V:)dy = 1). Similarly, we define the conditional probability of given and 3/ as py (y xx (x) = p(yl=)
probability of y given V/ as py (y) [D)) == p(y)
p(): if E Vi, otherwise 0. For a label C E [K], we define the conditional probability of subset Ve given x and the union of disjoint subsets D as Pa(c 2; (Di))e(K)) = (X,z). We regard this as a probability function of labels over [K] as EeE[K] Pa(c1 z; (Dt)se(k)) = 1. Last, we construct a linear classifier on learned representations. Given the disjoint subsets (J%))e(k) and the components of similarity g(æ, v) = fx(x)) fy(w)/T, we define 7 (xx) = W (x( + b, with weight W == [t01, W2 ... Wx] E BRdx th [() E R., and bias b = IIn (Di), In (r2),,
Epy (vDi))
In Py(x))] E Rd.
Now, we show an upper bound on the excess risk of the downstream classification when we obtain encoders that achieve the optimal similarity of the symmetric InfoNCE.
Theorem 4.2. Let (Vi)eEjk) be any choice of disjoint subsets in V. Assume that g" (z. v) ==
7. (x) ( In p(z.0) + const holds for any x E suppp(#>) S X and any y V. Then,
p(x)p())
[DK (Po(C) 2²) Po(C | a;; (1)))))))
Coup (hr)
S
E p(a)
+ E [DKL(Py (Y (1)) II (Y x,,o,)
(3)
p(a,o)
We defer the proof to Appendix
B.1
Remark. The first term in RHS of Eq. becomes zero when, for any c and xr the conditional probability (r,))=)))) is proportional to the conditional probability of label Pc(cc)x). The second term in RHS becomes zero when is independent of x given a prior knowledge that is in Ve. Considering the prompt ensembling in zero-shot classifications (Radford al.) 2021] and the properties of text data, we claim that there exist subsets ()%)sEtK1 that satisfy most of those conditions. To construct a classifier in zero-shot classification, [Radford et al. (202T) proposed ensembling embeddings of prompt templates such as "a photo of { and "an example of a ()", where the brackets are replaced with the labels such as "dog" and "cat". Since the set of prompts for each label is generated simply by inserting words representing the label into templates, the probability of each set should be roughly proportional to the probability of the label. In addition, prompt templates lack most of the information specific to images, each prompt in the set can be considered more or less independent of images. Assuming these properties of the text data domain, the excess risk of the linear classifier close to zero when trained encoders achieve the optimal similarity, which is the pointwise mutual information.
4.3 EXCESS RISK ANALYSIS VIA THE GAP FROM THE POINTWISE MUTUAL INFORMATION
We have observed that a similarity equal to the pointwise mutual information (up to a constant) leads a small excess risk of linear classifiers on the downstream classification. However, an actual similarity g(2, y) obtained in pretraining is possibly different from g (z., y) because of the non-convexity of the optimization problem and the insufficient representational capability of the class of similarity, { (Ix,)) y) + fxxx)) fy(u)/T fx(x), fy(v) E TRd, 7 E Ryo}_ To consider the effect of the gap the similarity, we decompose the risk of the downstream task as follows:
Csup(h Caup = (Coup Lsup(he")) ))+ +
"Lu
(4)
The second term in RHS of Eq. is already bounded by Theorem 4.2 have the following bound.
Regarding the first term,
Lemma 4.3. Assume that, there exists A such that [g(ar, -)- and all y E supp ply). Then, it holds that (Cmp((h) (ho) Laup (ho" < 24.
gg (x, v)1 < A for all x E supp
We defer the proof to Appendix B.2 From Theorem 4.2 Lemma
and the fact that eiew BERK Caup(W fx(+) + b) S Laup (h2), we have the following result.
Theorem 4.4. Assume that there exist K disjoint subsets Y (i E [K]) S y such that DKL. (po(C) 1x) pp(CC x; (Dilie(x)) < En, and (RI (1)) || py (Y | ax vc)) S ez, for
5

---

Published as a conference paper at ICLR 2025
all E supp p(x), for all [K], and for some non-negative constants E2, 2 0. Assume that the uniform approximation error of the optimization problem arg min, (xCE(g) bounded by constant A 2 0, i.e., (g(z, y) - 9" (z. y) & for all x E supp s()) and y E supp ply). Then, holds that
min WER BERK
Suu (W fx()) + b) - Ssup (h") E + 2A.
(5)
Remark. A indicates the gap between actually obtained similarity g(z, y) and the optimal similarity gz y). which is the pointwise mutual information. Theorem 4.4 that the approximation error of the optimal similarity in pretraining may degrade the performance of downstream classifications.
AUGMENTED SIMILARITY BY WEIGHTED POINT SETS
We have observed that the optimal similarity of symmetric InfoNCE for pretraining leads to small excess risk on downstream classifications. Here, a question arises: "To what extent can the class of similarity approximate the pointwise mutual information?" In this section, we show limitation of the typical similarity that is commonly utilized in CLIP. To overcome the issue, we propose new class of similarities and show a theoretical guarantee of the approximation capability of the proposed class.
5.1 LIMITATION OF THE INNER-PRODUCT SIMILARITY IN FINITE DIMENSIONAL. SPACES
Consider a d-dimensional feature space. We assume there are ((> d + 1) pairs of samples, (z1,31),... y1)..... (IN, yN) E X V. We define Zx Zy E R the concatenation of features of samples as Zx == [fx(x)),... f...xxxxxx and Zy == (fn())),..
fy(UN) }]. During pretraining with the symmetric InfoNCE, the similarity matrix Z Zy fit to the optimal similarity matrix G E N
up to a constant 1 E R. where Gi- = In p(..,))) Regarding the gap A to the optimal similarity. p(z.)(())))
Zy);; r Giyl However, it also holds that rank _ Z + IJ) < d 1, where J E RNNN is the matrix in which all entries are 1 (See Proposition C.I). Thus, the rank G is > + 1, there exists a certain error of the approximation of G. In other words, to completely capture the structure of the pointwise mutual information, the dimension of feature dl is required be more than the number of intrinsic instances in the data space, which is infeasible in real-world scenarios.
holds that A 2 supzesupp (Esupp p(v) (x. y) g (x(r,)) 2 sup,,
5.2 AUGMENTED SIMILARITY BY A NONLINEAR KERNEL AND WEIGHTED POINT SETS
Increasing the dimension of the feature is the simplest way to enhance the capability of the similarity. However, this often requires larger deep neural network model, which leads heavier computation both in the contrastive learning phase and in the downstream tasks. As an alternative approach, we propose enriching the class of similarity by using nonlinear kernel function and weighted point sets (namely, sets of pair of weight and point). Figure [ shows the overview of the proposed method. We replace the similarity in the symmetric InfoNCE with a similarity between two weighted point sets produced by encoders.
Following CLIP, we use two encoders that transform inputs from each modality. Instead of one vector in latent space, the encoders are modified to produce a weighted point set, namely, set of M pairs of weight and vector: ((w,, ())) LE[M] where w, E IR and v; E IRd for each E [M]. We define
the similarity of two weighted point sets, { (w/)() of ( se(Arce)] and {
(wwf") o =E(arr)))
(containing M (x) and M0') pairs of weight and vector, respectively), with a kernel function k(;-)): d x Rdd R. as follows:
(2) o (1( (1(l0) 10,0
Le(pronny {{ (w))0) Meswaro
(2 (Mroo = Σ wi of og))) (6)
(.j
This similarity can be regarded as the inner product of high-dimensional representers of a lin- combination of Dirac measures (Muandet al. 2017) Ens wi m
(x), ( (09)

---

Published as a conference paper at ICLR 2025
Image input 2:
Image encoder
Similarity
Weighted point sets
Embeddings
m
(w PP))) Mo0
Text input
Text encoder
Overview of proposed method. Each encoder produces weighted point set from each input. The encoders are optimized with the symmetric InfoNCE using the similarity matrix.
Figure 1:
Here, (, denotes the inner product of the repro- ducing kernel Hilbert space (RKHS) associated with k. In the following theorem, we show that our proposed similarity based on weighted point sets consistently achieves the optimal similarity.
E. ). E, w
Theorem 5.1. Assume that Assumption C.2 holds. Define a function g as Eg. @with bounded co- universal kernel k; Ra x Rd + R. Then, for any >> O, there exist positive integers, M M e
and maps, fx x. 1> {(") (X) (x))
(x))
ke telasco] and fy : + W (2)
(V)
je se(mron such that
((x(1t) In p(x, v) p(r)p())
sup zésupp p(s) p(u)
<e
The proof and Assumption C.2 are provided in Section C.2 The definition of co-universal kernel is deferred Definition (refer to Srriperumbuur al. (01T)). For example, the Gaussian kernel k(u, v) exp u and the inverse multiquadric (IMQ) kernel k(u,v)
are cn-universal (Sriperumbudur et 201T.
Remark. Theorem 5.] ensures that the proposed class of similarity is capable of approximating the pointwise mutual information arbitrary precision. Unlike the typical class of similarity discussed in Section [. Assumption C.2 does not require the dimension d proportional to the number of intrinsic instances. Instead, it requires d larger than or equal to the intrinsic dimensions of subspaces of æ e X and y E y that have dependency fairly mild because the manifold hypothesis (Bengio et al.) (013) commonly assumed. Although increasing (x) and MD) also leads to heavy computation, at least provides different approach to augmenting representation models than just increasing the number of feature dimensions.
each other. However,
claim that this assumption on is
5.3 IMPLEMENTATION
To produce a weighted point set from each input, we utilize the structure of transformers, without any significant change the model size or computation time (Fig. 2 We use Vision Transformer (Dosovitskiy et al., 202T) for the image encoder and Transformer Vaswani et al., 2017) for the text encoder. A typical Vision Transformer takes projected patches an image and the special token [CLS], applies attention layers and the last projection layer to the token sequence, and outputs the vector the position of the [CLS] token. To output additional weights, we add a projection layer for weights in parallel with that for vectors. Moreover, to output sets of weights and vectors, our image encoder outputs all resultant vectors. In the same way, we modify the text encoder to output all resultant weights and vectors instead of just the vector at the position of special token [EOS]. In addition, we modify the special tokens the text encoder for padding. [PAD], to be dependent on its relative position to the [EOS] in order to avoid repeating the same tokens. More specifically, separate padding tokens [PAD1], [PAD2], etc., are appended after the [EOS] until the token sequences reach a fixed length. A learnable embedding is independently initialized for each token.
7

---

Published as a conference paper at ICLR 2025
Weighted point set for input image
Weighted paint set for input text
weight We
Image encoder
Text encoder
t [CLS]
[SOS]
lonely
[EOS]
[PAD1]
[PADm]
Image
Split into patches
Tokenization
"onelyy kitten seeks comfort in the form of cuddly cal"
Text
Figure 2: Proposed modification for encoders to produce a weighted point set. Encoders are modeled by Transformer. The encoders output all resultant vectors instead of just one vector at certain position.
For the kernel function, we opted to use a linear combination the linear kernel and nonlinear
kernel & with coefficients Q1, Ol2 Rzyi k(u, v) = 01 Toguuvton aykku, ( v). In preliminary experiments, found that when the model was trained only with nonlinear kernel (i.e., (a1, 022 (0. 1)) the symmetric InfoNCE loss did not decrease well nor converge. We consider this was possibly because of the gradient vanishing for points that were far away from each other. To avoid ((M (X) M (?)) times computation of the kernel, we use random Fourier features (RFFs) (Rahimi & Recht 2007) for approximating the nonlinear kernel. When the kernel À is chift-invariant, RFF approximates the kernel k(u, v) by the inner product of two D-dimensional vectors, ie. 2(u) T (vv) se k(u, u). z(v) I is constructed using random samples We E Rd and 3, E (t 1,..., D) from predefined distributions (see Appendix A. for details). By taking the weighted sum of RFFs, = Σ, w;,(t;), calculated from points in the point set, we obtain an embedding of the weighted point set. More rigorously, this can be regarded the embedding the RKHS of linear combination of Dirac measures where the RFF approximation is applied to obtain finite dimensional representations of the embeddings. In our implementation, we concatenate the weighted sum of points,
Σ; W;Us, and that of RFFs using coefficients a and a2 follows: [Varv" /a2zz We use it embedding of weighted point sets because we can obtain an unbiased estimator of the similarity in Eq. (6) by simply taking the inner product of embeddings:
Σ 1,
( W
(X)
1
~
Σ
wi
(2))
( (XT " (2))
an
(a (x)
VaiT Varzz()) (2)
(
worth noting that the dimension of RFFs can be changed between training and inference, which affects the kernel approximation error. For example, larger D can be used during pretraining to achieve lower variance approximation, and a smaller can be used during inference to reduce computational cost and memory usage.
6 EXPERIMENTS
6.1 PRETRAINING
To investigate the performance of the representation based on weighted point sets, Weighted Point Set Embedding (WPSE), we conducted experiments in which trained text-image representation model. We utilized Conceptual Captions 3M (CC3M) (Sharma et al. (0188) and Conceptual Captions 12M (CC12M) (Changpinyo al. 2021) datasets for pretraining. As the base architecture of the image encoder, we adopted VIT-B/I6 (Dosovitskiy et al. 2021). Following SLIP (Mu et al. 2022), used the smallest text Transformer model from CLIP. We modified the image encoder
8

---

Published as a conference paper at ICLR 2025
Table 1: Zero-shot classification performance. We report the per-class accuracy (%) on
Caltech-101, Aircraft, Flowers, and Pets. On other datascts, we report the top-1 accuracy (%).
Average
ImageNel
CIFAR-10
Food-101
Caloch-tt
Aircaft
EurOAAT
DTD
Pets
Model
CLIP
25.03
19.94
22.48
75.24
13.05
47.20
1.38
13.11
10.40
13.56
14.62
CC3M
WPSE Gaussian
26.75
21.20
59.95
23.58
80.61
14.56
51.18
1.49
1.35
12.60
19.98
13:40
13.60
34.16
WPSE IMQ
27.04
21.36
61.22
25.91
81.64
13.17
50.15
1.41
1.84
12.14
22.02
13.69
13.88
33.05
CLIP
43.78
39.15
74.17
42.98
90.91
47.96
73.58
21.94
2.01
29.71
22.24
22.45
52.29
49.72
WPSE Caussian
21.17
49.57
WPSE IMQ
45.71
39.26
80.31
47.53
91.83
51.82
73.54
21.92
1.62
29.53
28.36
21.62
57.31
49.54
Table 2: Linear classification performance. We report the mean per-class accuracy (%) on Caltech-101, Aircraft, Flowers, and Pets. On other datasets, report the top-1 accuracy (%).
Averge
ImageNet
CIFAR-100
STL-10
Food 101
Calistt100
Aircraft
Flowers
EwoSAT
SUN397
Cars
DTD
Pets
Model
CLIP
67.00
51.42
85.51
61.87
91.71
61.71
79.24
27.27
31.81
86.67
93.82
63.19
66.48
67.35
79.97
CC3M
WPSE TMQ
68.23
56.77
85.87
63.48
92.06
64.12
80.78
27.16
33.98
87.47
93.72
63.14
69
99.06
69.35
CLIP (bel)
72.14
58.33
87.90
70.05
92.64
66.07
82.49
39.46
44,96
91.48
96.02
67.02
71.86
69.56
WPSF Gaussian (bef)
73.77
61.19
87.94
70.36
92.70
69.37
84.03
44.50
47.93
92.10
95.86
67.71
74.10
71.21
WPSE IMQ (bef)
73.81
61.02
88.44
70.10
92.68
68
68.84
84.39
47.66
95.92
67.61
75.94
71.10
CLIIP
77.89
71.75
64.53
92.50
94.32
81.71
76.55
WPSE Goussian
79.08
67.83
91.72
73.06
96.46
79.49
89.18
65.23
44.53
92.09
94,58
72.93
83,56
77.42
CC12M
WPSE IMQ
78.90
67.11
91.14
72.59
96.48
78.69
88.
66.32
44.17
92.61
94.70
72.55
83.21
77.31
CLIP (bof)
81.03
69.15
92.04
75.99
40
80.13
90.09
70.86
53.72
94.85
96.64
74.89
81.81
77.75
WPSE Gaussian (bef)
82.52
96.30
78.62
WPSE IMQ (bel)
82.71
70.90
93.04
77.27
96.60
81.58
91.06
75.53
58.10
95.60
96.28
75.43
85.65
78.19
and the text encoder to produce weighted point sets (as explained in Section 5.3) As nonlinear kernel k. we used the Gaussian kernel and the IMQ kernel. We performed hy hyperparameter search over a (for the Gaussian kernel) and c (for the IMQ kernel) in the range of (0.5, 0.75, 1.0]. We also ran a hyperparameter search on the coefficients (a1, (22) for combination kernels. We searched (01, .22 = (0.667, 0.333), (0.6, 0.4), (0.5, 0.5). (0.4, 0.6). (0.333, 0.667). In the tables, we report the performance of the best model from the hyperparameter search. During the pretraining, the dimension D of RFFs to 1024. For each batch, new w, and Br for RFFs were sampled during the pretraining. For comparison, also trained typical CLIP models from scratch. For more training details, see Appendix A.2
set
6.2 ZERO-SHOW TRANSFER
We evaluated the zero-shot classification performance on the following 13 benchmark datasets:
ImageNet (Russakovsky et al., 2015), CIFAR-I0 (Krizhevsky 2009), CIFAR-100 (Krizhevsky 2009).
STL-10 (Coates et al. 2011). Food-101 (Bossard et al.) 2014), Caltech-101 (Fei-Fei et al. 2006).
Stanford Cars (Krause et al. 2013). FGVC Aireraft Maji et al. 2013). Oxford Flowers Nilsback
& Zisserman 2008). EuroSAT (Helber et al., 2019), Describable Textures Dataset (DTD) (Cimpoi
al. 2014), Oxford Pets (Parkhi et al. 2012). and SUN397 (Xiao al. 2010). Following SLIP (Mu
et al. 2022). we adopted prompt ensembling and utilized prompts provided by SLIP for each dataset. We set the dimension D of RFFs to 512. W. and B, for RFFs were fixed before the evaluation. To investigate the effect of the randomness of RFFs, we performed five evaluations for the models that use RFFs. Table I lists the zero-shot classification results, where the results of models using RFFs have been averaged. Additionally, Table the Appendix shows the standard deviation. As these findings show, the proposed method outperformed CLIP on average. addition, the randomness of RFFs did not have significant impact the overall performance.

---

Published as a conference paper at ICLR 2025
6.3 LINEAR CLASSIFICATION
We also performed the linear classification evaluation where we trained linear classifiers on the embedding vectors obtained by frozen pretrained image encoders. We used the same 13 benchmarks as the zero-shot classification. To extract embeddings for training linear classifiers, we used two different settings. In the first setting, we used the embeddings that were used for computation of the similarity in the symmetric InfoNCE. We set D for RFFs to 512. wy and 6e were fixed before the evaluation. Based on the robustness of the RFFs shown Table 5 we did not evaluate multiple settings of Wi and 3g. In the second setting, following common practice (Chen et al.) 2021), we used the intermediate latent vectors just before the last projection layer of the image encoder. We denote this setting as "(bef)" in tables. For our WPSE models, weighted sum of the latent vectors with weights in the output weighted point set was used, and no RFF was used.
We basically followed the evaluation procedure in Fürst al. (2022). We used logistic regression classifier with an L-BFGS optimizer (Liu & Nocedal 1989) and the maximum number of itcration of 1000, We utilized the implementation from cuML (Raschka et al. 2020). For hyperparameter tuning of the L. regularization cost, we followed the protocol of CLIP (Radford et al. (021). We validation split of each dataset. For datasets that do not provide an official validation split, we randomly split the training dataset into training and validation splits. After the hyperparameter was determined, we trained a classifier on the combination of training and validation splits and report its performance on the test split. Table 2litts the linear classification results. Overall, our proposed method outperformed CLIP on average.
ran hyperparameter sweeps over C' E [10-6 10 with parametric binary search
6.4 ABLATION STUDY
To investigate the effectiveness of our similar- ity, we trained two variant models that output weighted point sets on CC3M. One model out- puts weighted point sets but the all weights are positive: Wi 2 O. We used a function 100Sigmoid(//100) as the last activation for weights in encoders. We denote this model as WPSE with postive weights. The other model also outputs weighted point sets but the simi- larity of weighted point sets are calculated only with linear kernel, i.c., the coefficients (a1, a2) are set to 0). We denote this model as WPSE Linear. We also trained the model with the coefficients (at, ax2) = (0. 1). specifically only with the nonlinear kernel. However, the training of this model failed due to a NaN loss error. Table shows the average performance of zero-shot classification and linear classification on the 13 benchmark datasets. For WPSE with positive weights, we used the same parameters for the combination kernel. This indicates that negative weights are crucial for a good performance. (We did not perform linear classifications for WPSE with positive weights.) In comparison to WPSE Linear, it indicates the superiority of the use of non-linear kernel in the linear classification tasks. We also show the result of ablation study using CC12M in Appendix A.4
Table 3: Ablation study. Models are trained on CC3M. Except for WPSE Linear, IMQ kernel was used.
Model
Zero-shot Linear
WPSE
27.04
68.23
WPSE with positive weights
4.22
WPSE Linear
27.25
67.40
7 CONCLUSION
We proposed a multimodal representation learning with weighted point sets. In our method, each input is transformed by an encoder into a weighted point set representation. The similarity between two weighted point sets is calculated with a kernel function that defines the similarity of two points. We also showed the theoretical benefits of using our representation and similarity. We highlighted that the optimal similarity of the symmetric InfoNCE is represented by the pointwise mutual information and showed that we can construct a linear classifier close to the optimal classifier of downstream tasks that is possibly nonlinear when the optimal similarity is obtained. In addition, we clarified the effect on the performance of downstream tasks caused by the deviation of the obtained similarity from the pointwise mutual information, and explained that the deviation of the similarity can be suppressed when using the proposed similarity based on weighted point sets, Experiments on text-image datasets demonstrated the superior performance of the proposed method compared to baselines.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
In conducting this research on representation learning models, are committed upholding ethical standards. Our work aims to contribute to machine learning research society by theoretical analysis of representation learning and enhancing the capability of representations. However, we recognize potential concerns of representation learning models, such as biases in training datasets, license issues of scraped datasets and harmful applications. We acknowledge that representation learning models can have significant impacts on society. Therefore, we commit ourselves to ensuring that our research activity positively contributes society while avoiding harm.
REPRODUCIBILITY STATEMENT
Detailed descriptions of our setup of the algorithm and experiments can be found in Section 5.3 [ and Appendix A Moreover, we release our code at htttttts///gtthi com/ sony wpse to ensure reproducibility.
ACKNOWLEDGEMENTS
Computational resource of AI Bridging Cloud Infrastructure (ABCT) provided National Institute of Advanced Industrial Science and Technology (AIST) was used. TS was partially supported by ISPS KAKENHI (20H00576) and JST CREST (JPMICR2015), We would like extend our thanks to Wei-Hsiang Liao and Bac Nguyen from Sony AI for the valuable feedback.
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, 68(3):337-404, 1950.
Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning. In International Conference on Artificial Intelligence and Statistics, pp. 7187-7209, 2022.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in Neural Information Processing Systems, 32:15535- 15545, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1798)1179 2013.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 1ood-ing discriminative compo- nents with random forests. In European Conference Computer Vision, pp.
2014.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding transferable representation learning and zero-shot transfer in CLIP. International Conference on Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2025
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metrie discriminatively, with application to face verification. In IEEE Computer Sociery Conference on Computer Vision and Pattern Recognition, pp. 539-539--6 2005.
Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In International Conference Artificial Intelligence and Statistics, pp. 215-223, 2011.
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam. Hyperbolic image-text representations. In International Conference on Machine Learning, pp. 7694-7731, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5, 2023.
Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 28(4):594-611, 2006.
Andreas Fürst, Elisabeth Rumetshofer, Johannes Lehner, Viet T. Tran, Fei Tang, Hubert Ramsauer, David Kreil, Michael Kopp. Günter Klambauer, Angela Bitto, and Sepp Hochreiter. Cloob: Modern hopfield networks with infoloob outperform clip. Advances in Neural Information Processing Systems, 21220400-20463 2022.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15199 15190, 2023.
Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: A survey, IEEE Access, 7:63373-63394 2019.
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image. text and audio. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 976-980, 2022.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-5011, 2021.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the generalization of contrastive self-supervised learning. In International Conference on Learning Representations, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and bision-languee representation learning with noisy text supervision. In International Conference Machine Learning, pp. 4904-4916, 2021.
12

---

Published as a conference paper at ICLR 2025
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In IEEE International Conference on Computer Vision Workshops, pp. 554-561, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Yazhe Li, Roman Pogodin, Danica J Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. Advances Neural Information Processing Systems, 34: 2021.
Yan-Bo Bo Lin, Jie Lei, Mohit Bansal, and Gedas Bertasius. Eclipse: Efficient long-range video retrieval using sight and sound. In European Conference on Computer Vision, pp. 413-430, 2022.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming. 45(1 (5(1):503-528, 1989.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. European Conference on Computer Vision, pp. 529-5499-444 2022.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Schölkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends@ in Machine Learning, 10(1-2)11-14 2017.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, pp. 722-729, 2008.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:/807.03744 2018
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEE 2012.
Conference on Computer Vision and Patlern Recognition, pp.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8026-8037, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. 8763, 2021.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in Neural Information Processing Systems, 20: 111117771184 2007.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Sebastian Raschka, Joshua Patterson, and Corey Nolet. Machine learning in python: Main develop- ments and technology trends in data science, machine learning, and artificial intelligence. arXiv preprint arXiv:2002.048833 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1 115:211-252 2015.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Confer- ence on Machine Learning, pp. 5628-5637 2019.
13

---

Published as a conference paper at ICLR 2025
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2566-2565, 2018.
Zhenmei Shi, Jiefeng Chen, Kunyang Li, Jayaram Raghuram, Xi Wu, Yingyu Liang, and Somesh Jha. The trade-off between universality and label efficiency representations from contrastive learning. In International Conference on Learning Representations, 2023.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in Neural Information Processing Systems, 29: 187-11865, 2016.
Bharath K Sriperumbudur, Kenji Fukumizu, and Gert RG Lanckriet. Universality, characteristic kernels and rkhs embedding of measures. Journal of Machine Learning Research, 12:2389-24115 2011.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision, pp. 776-794, 2020.
Christopher Tosh, Akshay Krishnamurthy. and Daniel Hsu. Contrastive learning, multi-view re- dundancy, and linear models. International Conference on Algorithmic Learning Theory, pp. 1179-1206, 2021.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. International Conference on Learning Representations, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30:5998-6008. 2017.
Hiroki Waida, Yuichiro Wada, Léo Andéol, Takumi Nakagawa, Yuhui Zhang, and Takafumi Kanamori. Towards understanding the mechanism of contrastive learning via similarity structure: A theoretical analysis. In Machine Learning and Knowledge Discovery in Databases: Research Track, pp. 709-727, 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference Machine Learning, pp. 9929-9939, 2020.
Yifci Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations, 2022.
Ho- Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4563 4567, 2022.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485-3492, 2010.
Runtian Zhai, Bingbin Liu, Andrej Risteski, J Zico Kolter, and Pradeep Kumar Ravikumar. Under- standing augmentation-basse self supervised representation learning via RKHS approximation and regression. In International Conference on Learning Representations, 2024.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Con- trastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, volume 182, pp. 2-25, 2022.
Yuhui Zhang, Yuichiro Wada, Hiroki Waida, Kaito Goto, Yusaku Hino, and Takafumi Kanamori. Deep clustering with a constraint for topological invariance based on symmetric infonce. Neural Computation, 35(7): 288-1339, 2023.
14

---

Published as a conference paper at ICLR 2025
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 12979-12990, 2021.
15

---

Published as a conference paper at ICLR 2025
Algorithm Symmetric InfoNCE loss with the similarity of weighted point sets
Require: an image encoder fxx text encoder fy, a batch of B paired images and texts {26, yo ga)her the distribution Pa associated with the shift-invariant kemnel k, coefficients on and o2, and temperature т.
1: b1 61
(2)
(2)
(2) (2))
bar (br()) bbr(())
fx(s)) for each b E [B]
11 61
(2)
w (0) 1,00
barco)) bar(o)) + (y0) for each E
3: T + Sari
(20)
(20
Whe
bb
4: To + - Whj 1b
(0 (2)
5: Draw Diiid. samples way wD from Pa-
Draw D i..dd samples ...... BD from Unif(O, 2T).
7: x/ < EEE MM(a
(x) (2), [wr) [3))
for each b E [B]
26 + E,- Why 2 on fwe {8)) for each E [B]
(00
9: Say (aro (
oy = = for each b, [B]
10: Compute the symmetric InfoNCE loss from the similarity matrix {Swa Tay-
A ADDITIONAL DETAILS OF IMPLEMENTATION AND EXPERIMENTS
A.1 IMPLEMENTATION
Random Fourier feature (RFF) Random Fourier feature (Rahimi & Recht 2007 is technique for reducing computational complexity of kernel methods. For a shift-invariant kernel k(u, v) k(u 10--0 on Rdd such that (()) 1. there exists probability distribution, Pw. of a random variable, R
that satisfies:
k(u - v) = E [2cos(wT Toel(wTee u 3) cos(w Tu 8)] w,g
where 3 R is sampled from uniform distribution, Unif(0, 2T], over [0, 2n.. Pw is given by the Fourier transform of k(u-v) Based on this fact,
construct an unbiased estimator of k(u, v) as follows. First, w, E d and 3, E IR (f = 1, D) are independently sampled from the distributions Pe and Unif[0,27]] respectively. Then, vector 2(v) E IR is constructed from Rd. (w)) {wr)err and {,,0
a (v [ (8.) V [oos(ww] ( v
Bi),..., cos(wwu + (
BD)]
Similarly, 2(u) is constructed from with the same [we)e and { 1-11-11-- Last, estimator of k(u,e) is obtained by taking the inner product of the vectors: E[z(u) 2(v) k(u, v). For the specific form of pa for Gaussian kernel and IMQ kernel, and further details, see Appendix C in Li
al. 202T). Algorithm shows pseudocode for computing our proposed similarity for symmetric InfoNCE.
modify Transformer encoders as follows. To stabilize training, we add an activation function, 100 tanh(-/100), after the projection layer for weights for restricting the range of weights. In preliminary experiments, we found that model parameters diverged during pretraining without it. Following CLIP, we apply L2- nomalization to points v; (i [M] in weighted point sets, and use an inverse temperature parameter - to scale the similarity of weighted point sets (Algorithm 1). In typical CLIP implementations. is calculated by an exponential activation as - = exp(0) with learnable parameter 0 and clipping to certain range, such as [1, 100). However, in preliminary experiments, we found that -1 increased rapidly to the maximum value the beginning of pretraining when we use the exponential activation and the weighted point set similarity, and that it harmed model performance. Therefore, we remove the exponential activation and use clipping is set [1, 100).
Model architecture In addition the modifications in Section 5.3
to scale the proposed similarity. The range for
16

---

Published as a conference paper at ICLR 2025
A.2 PRETRAINING
The text Transformer model we used is 12-layer 512-wide transformer with eight attention heads. We utilized a byte pair encoding (BPE) tokenizer with a vocabulary size of 49K and a maximum context length of 77. Based on the Transformer architectures, we set M(2), MO), and d of the weighted point sets to 197, 77, and 512, respectively. As data augmentation, images were randomly resized and cropped with scaling factor between 0.5 and and bicubic interpolation. Models were trained for 50 epochs CC3M and for 35 epochs CC12M. We set the batch size to 2048. We used the AdamW optimizer with a beta2 of 0.98 and cosine scheduling with a linear warmup in pretraining. We set the initial learning rate to 0.0005 and used weight decay of 0.5. We used the built-in automatic mixed precision library in Py Torch (Paszke et al. 2019).
A.3 CLASSIFICATION EVALUATIONS
Table 4: 13 datasets used for classification evaluations.
Datasct
Classes
Train
Val
Test
ImageNet
Russakovsky
2015
1000
1153051
128116
50000
CIFAR-
Krizhessky 2009
10
45000
5000
10000
CIFAR-
(Knzhevsky 2009
100
45000
5000
10000
STL.-10 Coales al. 2011
10
4500
500
8000
Food-101 Bossard et al.
2014
101
68175
7575
25250
Caltech-10h Fei-FFi et al.
2006
102
2754
306
6085
Stanford Cars (Krause el a 2013
196
7330
814
8041
FGVC Aircraft (Maui et al. 2013)
100
3334
3333
3333
Oxford Flowers (Nilsback & Zisserman
2008
102
1020
1020
6149
EuroSAT (Helber |019
9000
1000
5000
DTD (Cimpor et al. 2014)
1880
1880
1880
Oxford Pets (Parkhi ct al
2012
37
3312
368
3669
SUN397
Xiao al. 2010)
397
76129
10867
21758
The properties of the datasets we used the classification tasks are listed in Table 4 In Table we show the results of the same zero-shot classification presented in Section 6.2 but with the standard deviation included.
Table 5: Zero-shot classification performance.
CC3M
CC12M
Dataset
WPSE Gaussian
WPSE IMQ
WPSE Gaussian
WPSE IMQ
ImageNet
21.20 0.05
21.36 0.04
39.95 0.06
39.26 0.06
CIFAR-10
59.95 0.20
61.22 0.51
81.33 + 0.33
80.31 + 0.24
CIFAR-100
23.58 0.13
25.91 1 0.19
49.49 0.16
47.53 1 0.0S
STL-10
80.61 +: 0.37
81.64 0.25
9.02
91.83 0.13
Food-101
14.56 0.08
13.17 0.05
0.08
51.82 0.20
0.20
Caltech-101
51.18
0.10
7.254 0.220
Cars
0.02
1.41
0.114
2.192
Aircraft
1.12
1.84
2.09
1.62
Flowers
12.00
0.15
3.25
29.53
EuroSAT
19.98 0.18
22.02
23.28 0.36
28.36 0.38
DTD
13.40 +
0.11
21.17 0.23
21.62+0.22
Pets
13.60 0.18
13.88
6141101
57.31
SUN397
34.16 1 0.11
33.05 0015
49.57 0.11
49.29
A.4 ABLATION STUDY ON CC12M
17

---

Published as a conference paper at ICLR 2025
In this section, present the result of abla- tion study using CC12M. We trained two vari- ant models that output weighted point sets. One model is a WPSE Linear, which we described in Section 6.4 with the coefficients (01, 02)
Table 6: Ablation study. Models are trained on CC12M. Except for WPSE Linear, Gaussian kernel was used.
Model
Zero-shot Linear
(1, 0) and the similarity of weighted point sets calculated only with linear kernel. The other model has the coefficients (01, 02) (0. 1) and calculates the similarity of weighted point sets using only a nonlinear kernel. We denote this
WPSE
46.12
79.08
WPSE Nonlinear
44.05
70.99
WPSE Linear
45.87
78.61
model as WPSE Nonlinear. Additionally, we also trained a WPSE with postive weights with the
last sigmoid activation in the same manner described inSection 6.4 However, the training of this model failed due to a NaN loss. Table shows the average performance of zero-shot classification and linear classification linear kernel and a nonlinear kernel is beneficial for the performance.
the 13 benchmark dataset. This indicates that the combination of the
B PROOFS OF STATEMENTS IN SECTION
B.1 PROOF OF THEOREM 4.2
Proof. From the definition of (x)), the e-th entry of he (x) is calculated as follows:
h (z) = ( E. p(v))(.)
()))) fx(()) P(V())
(((())))) T (2(00) + II P(V.)
E
p(y))a)
(z,,,))) + In P()))
p(UD)))
In p(z,))
+ +Iv.( +T.
p()D))
p(x)p(y)
Since adding a constant to all entries of h(x) doesn't 't change the supervised loss Csup we consider T 0 for the sake of simplicity. The e-th entry of Te (x:) is further rearranged as follows:
he (x) = E
In
+ 1I P(())))))
P(y)
p(z)p(u)
p(z,w))(z))(() p(z)ply)p(x,)) Vo) plz, plz,y)/p(x,3d) p(v)/P()))
+ In P(x,)1)
E
+ In P().)
P(x))
p(u)).)
In Vc)
E p(u)).)
p(x)
In plyle, Vio)
E p(M(DL))
In P().)z)))
P(y)).))
In (Z)
DKL (py (YI))) py (Y)x
18

---

Published as a conference paper at ICLR 2025
Therefore, have
Caup (h3*)
(lup(h")
In P(c[z) - -ii5*()) In (E exp hor (1))2)
p(a,e)
In P(c/z) - In P(Diko)) DK.. (prr(Y))))) pr pr (x,
p(z,e)
In Σ P())aa) exp(- DEL ( (Y
In P(c]x) In P(V)(a)) DRL. (Dv (Y1))) py (r v.)) In Σ P(oilo))) ]
p(z,e)
[np((p>) In [(Vel()):) DEL (py (YI)) py (Y P()).)))
p(z,e)
In P(ca)) + DEL (py (V())) || pr (Y lo, (r,)
E p(z,e)
P(D))))/P(((z))
E p(z)
DKL (Po(Clx) Po(C x; (DWset))) + p(a.c)
[KL (Y py (Y)x)))))
Here, the inequality holds by the monotonicity of In(.), the non-negativity of P()))ar,, and the non-negativity of KL. divergence.
B.2 PROOF OF LEMMA 4
Proof. For every i E [K], it holds that
hv(x), h1" (x))
E
(9(x,
pptl)?)
E
[
Let <e)2) denote the logarithm of the c-th entry of the softmax function, i.e., <c(2) = In
21 ex
(x))) ΣK (xx))(x))
exph (x),
Lsmup(h Louppp (T19))
=
E p(z,c)
+ In
Σ exp T10 (x)
Se (h((z)) + sc h (x))
x p(z,c)
E
(10)
<e(2) is differentiable function with respect 2, and the partial derivative is given as follows:
Ose
=
Oze
Σ Σ
o=i
dc.
-ea
for
+
Zzg
ΣR
ex
By the mean value theorem, there exists on the line segment between he (x) and he (x) such that
(h)(z)) 5o hh9" (z)) V<c(E) (-1"(( h (x))
VVc(E)
19

---

Published as a conference paper at ICLR 2025
Therefore, have
- (h(()))) + So (n (h0'(z) (x))
V<oole (- (1))())+
he (1))
E p(2,0)
E
pla,c)
Usc (5)
)
E p(z,e)
ho (z)
Ozs
E p(r.0)
[2A]
2A.
(11)
Here, the first inequality holds by Hölder's inequality. At the second inequality,
Eijo e
Σ
(5)
2.
D
E5. 2i=1
K et
Combining Eq. 10
finishes the proof.
C PROOFS OF STATEMENTS IN SECTION 5
C.I LIMITATION OF THE BILINEAR SIMILARITY
Proposition C.1. Let A, B E RdxM M. and c. E R. Let J E MxM denote the matrix in which all entries are 1. Then, have rank(A" cJ) s 1+1.
Proof. We define A, B E Rdd+1)xM as follows:
A
B=
Then, we have ABB B A B - cJ. Since rank A S d+ and rank < d 1, the statement holds.
C.2 REPRESENTATIONAL CAPABILITY OF THE SIMILARITY BETWEEN WEIGHTED POINT SETS
We denote (joint) probability density functions of random variables by using their corresponding letters. For example, we denote the joint probability density function of the random variables X, Y and the probability density function of X as Px, and px. respectively.
We impose the following assumptions on the generation process of random variables X E X and Y E y.
Assumption C.2 (Generation process). There exist random variables X, Ý Rd. z Rdx and z E d that satisfy the following conditions.
(a) (X, Y), (xx), and Z are mutually independent.
(b) There exist continuous bijective mappings hx Rd Rdxx - X and by: Rd Rdy y such that X (x ((z) and Y hy(Y (Y, z(O)).
(c) The support sxp C Rd Rd of the distribution Px, compact.
X and y is an (d) The pointwise mutual information PMIx (MIx,y(T, (ā, = (x())pp())
(a.)(
L-Lipschitz function on suppP X supp py.
The second assumption means that data samples, X and Y, are generated from low-dimensional latent variables, (X, Z ((x) and (Y, 2(0), respectively. The first assumption means that dependency between X and Y stems only from X and Y. and that xX) and z are latent variables specific to the domain X' and Y, respectively. From the first and second assumptions, it follows that there exists
20

---

Published as a conference paper at ICLR 2025
1-to-1 I correspondence between (I, v) E X' y' and (ã, V, 2,,, 200) E
R
X
Rd X
and that (x.y(2.)) PR.y (2.
(= 2200) (= (=(33)
(2.5(E.8)
(x(2))
P. ( (200))p (5) (2)) (=137) Px (a)ppy((
To prove Theorem 5.1
the following statements.
Proposition C.3 Aronszzaj 1950; Sriperumbudur et al. 2011)). Let X be a topological space and let H be reproducing kernel Hilbert space of the functions on X with k: X X - R as its reproducing kernel. Then,
Σ 23)(, JEIN
n iJE [n]} R. (ay 1 [n]] CX
dense in H. Lemma C.4. Let X be topological space and let H. be reproducing kernel Hilbert space of the functions X with bounded kernel k: X X + R. Let supzex k(r, S For any f.g H. if [| -sllx < then || olloo /Kk.
Proof. For any x X. If(z) g(x)| = (l(2, 9/n lIxllf gllx < VKE.
Definition C.5 (co-universal, (Sriperumbudur et al., 2011)) bounded kernel, & with k(-,x) C,(X), Vo E X on a locally compact Hausdorff space X, is said to be co-universal if the RKHS, H induced by is dense in (X) w.r.t. the uniform norm. Le., for every function g E Co(X) and all e > 0, there exists an f H such that ||f -9lo0 S..
We now present the proof of Theorem 5.1
Proof of Theorem 5.7 First, we fix 6 >> 0. We prove the statement by explicitly constructing (x)), MMO) fx. and fy that satisfy 7
From (b) of Assumption C.2 there exist continuous inverse functions of hx and by. Consider the following restrictions of the functions hx² and hy² : for hx(ā, x(x)) and hy 2 =0)), it holds that
5 hx x (z).
2 (2) hx z(0) (x),
ÿ = hy l (V),
20) hr 12)0
Then, from (a) of Assumption
C.2
it follows that
(2 (2)) ( (2 (2))
Px.y (x.v)
px (x()))r())
Px(2)p2(x (x))) pr (00g(0))22 (0)p2:>> (2(2))
(ū,
px(#)pp(0) (ū)
Px,v (hx (z) (x))
(12)
px ( Ix (z)) py (h y (v))
To avoid complicated notations, we simply denote h (z) as ((x) and hy x as (yy) in the following.
From (c) of Assumption C.2 Proposition Lemma C.4 and the definition of the co- univerral
kernel, for any fixed j E supp Py. there exist M E N.
E R j E [M]. and {ūz Rd
[M]} such that, for any è supp
PMT R (t.0)
Σ
cgk(e,
<
(13)
JE[M]
21

---

Published as a conference paper at ICLR 2025
We denote such M, cj. and fly as (ū). (ū) and i, (t). respectively.
Meanwhile, we define (U) C I& as the open ball of radius r and center j IRd. From (c) of Assumption C.2 the support of py is compact. Thus, for any E > O, there exist J N and points 51, D2." ... ,NJ E IRd such that supppy S S Be/(2)) (ūj)- Given such Dj (i [J]). we define X(i) for S as one of the points, Dj (1 E [J]) that satisfies ý BB/((2)) (ūz). From (d) of
Assumption C.2 it holds that, for any (ž. 17) E S
PMI x x PMI
x v (2, x(0x 2
(14)
Now, ready construct desirable M(x), .MO), and fy- Let M(x) and MO)
maxje(J] Mūū)) We define fy : + { (1050) ) je[Moon]
0) cx((((())))
for 1 S S M(x((())) for M(x((())) <j & M for 1 1 M(x(()))
" ()) 0 0 = (x(((())
For oy with such that M(x((())) <jS MQ), we can choose any point in Rd. We define fx as fx(x) ((wi,,))) = ((1, }. Then, for every (I, suppyx,y X x X
())
In Px, (x, ((fx(())
px (x(x))p(1) (v)
ar(x) ar(3))
Σ
(2) (2)
(2) (e00)) klu!
w w
< PMIx,y(E(z)) ((0) - PM (o(z), x(((1)))
M00 M
Σ
(2) (
wi
1=1 3=1
M(X((())
5
(M1g -
Σ
9 (((()))))((())))))))))))))))
2
2
Here, the first inequality holds by the triangle inequality. The second inequality holds from Eq. 14 and the definitions of fx and fy- The third inequality holds from Eq. 13
22


Reviews:
[{"review_id": "m5uMmLv8DB", "reviewer": "Reviewer_Sno6", "summary": "The paper presents a new understanding of the symmetric contrastive loss InfoNCE of CLIP, and clarify the point-wise mutual information is the optimal similarity that minimizes symmetric InfoNCE., then gives an upper bound of excess risk on downstream classification tasks of representations when the optimal similarity is achieved. Then, the paper proposes similarity based on weighted point clouds to consistently achieves the point-wise mutual information and the corresponding implementation of the proposed similarity.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper starts from a new understanding of contrastive loss, and proposes similarity measure that approximates the optimal formulation, which is theoretically well-founded.\n\n2. The paper combines theory with practice and has demonstrated the effectiveness of the proposed similarity through experiments.", "weaknesses": "1. The innovation is limited. Some core concepts of the paper, such as suboptimality decomposition, are referenced from existing work.\n\n2. The role of weighted point cloud seems to be overstated. The paper did not use concepts related to point clouds, such as neighborhoods, connectivity. Only the “weight” is used in the weighted similarity aggregation.\n\n3. Some concepts or expressions need further clarification. For example, 1. What exactly does the similarity structure (line 41) mean? 2. Why is the rank of G greater than d+1, there exists a certain error of the approximation of G? (lines 309-310) 3. The assumption C.2 seems strong, can you provide a specific explanation based on practical examples? 4. The equation 3 has empty before ||, which seems a typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pLDFCdX26F", "reviewer": "Reviewer_UjLH", "summary": "The authors presented an analysis of the contrastive loss of CLIP that gave insight on the optimal similarity measure for this loss. Furthermore, inspired by their analysis of the optimal loss, the authors proposed a new similarity measure that achives optimal similarity values, with reasonable computational cost.", "questions": "It would be interesting to have an analysis of:\n\n- The effective rank of $G$ to clarify whether the stated deficiency of inner-product similarity is substantial. For instance, an analysis of the singular values of $G$ could be of value, since it would directly relate the rank disparity to the L2 error.\n\n- The effectiveness of linear-only kernels in addition to the experiments already in place.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an alternative to the inner-product similarity through the use of weighted sums of kernel function evaluations. As such, the method benefits from the kernel trick: whereas the simple inner-product similarity may require a high-dimensionality in feature space, the use of kernels reduces the dimensionality of the feature space and shortens the amount of calculations required. Conversely, the method allows for greater performance in downstream tasks for the same amount of computation, by approsimating more closely the (theoretical) optimal similarity measure.", "weaknesses": "1. Inner-product similarity analysis\n\n     A key point in the argument for the superiority of the authors' method over the standard similarity measure of CLIP is given in section 5.1, regarding the approximation error between the inner-product similarity matrix and the optimal similarity matrix $G$. There is an assumption that the number of samples $N$ is greater than the dimensionality $d$ of the feature space, thus the rank of the inner-product similarity matrix is smaller than the rank of $G$, implying the existence of approximation error. However, it is not clear from the paper that the approximation error is indeed deleterious, since the effective rank of $G$ could be substantially smaller than $N$ - and the effectiveness of CLIP suggests that this may indeed be the case. \n     Another evidence in this direction is the fact that the authors had to use a combination of a linear kernel with a nonlinear kernel in their approach, and their ablation study showed that the linear-only kernel has comparable performance (by exceeding in the zero-shot classification task, and underperforming in the linear classification task, and in both cases by a small margin that is not evaluated in regards to its statistical significance).\n\n2. Need for linear-only and nonlinear-only results\n\n     The hyperparameter options for $(\\alpha_1, \\alpha_2)$ do not include a linear only $(1, 0)$ and nonlinear-only $(0,1)$ option. Given the ablation results, such choices were desirable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X3JvkiZ5G9", "reviewer": "Reviewer_2vyS", "summary": "The paper proposes to replace the point-wise contrastive loss of CLIP with a dense contrastive loss calculated by using the contrastive loss on top of the average pair-wise similarity between dense features (the features prior to the final pooling layer of encoders). The authors show that this leads to better results (using the same evaluation as CLIP). The method is rigorously motivated by analyzing excess risk relative to the optimal downstream classifier head.", "questions": "None.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The introduction is especially well-written. I appreciated the motivation that many-to-many correspondences are not adequately represented by a single CLIP embedding.\n\n(2) The theory is nice (but Eq. 3 has typos).", "weaknesses": "(1) It is unclear how this method differs from dense contrastive loses. (For example: https://openaccess.thecvf.com/content/CVPR2021/papers/Wang_Dense_Contrastive_Learning_for_Self-Supervised_Visual_Pre-Training_CVPR_2021_paper.pdf)\n\n(2) Computational cost. The contrastive loss is already quite computationally expensive (square of mini-batch size if not using global hard negative mining). Now we multiply that cost by the square of number of feature vectors. \n\n(3) Practically speaking (not considering the theory), there is not much innovation in this paper. Similarity metrics over pairs of sets of features are well-established. For example: ColBERT (https://arxiv.org/pdf/2004.12832). The extension from a pure NLP setting to image-text is straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)