Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric

Authors: Toshimitsu Uesaka, Taiji Suzuki, Yuhta Takida, Chieh-Hsin Lai, Naoki Murata, Yuki Mitsufuji
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: contrastive learning, representation learning, multimodal representation learning, theoretical analysis, InfoNCE, pointwise mutual information

Abstract:
In typical multimodal contrastive learning, such as CLIP, encoders produce one
point in the latent representation space for each input. However, one-point representation
has difficulty in capturing the relationship and the similarity structure of a
huge amount of instances in the real world. For richer classes of the similarity, we
propose the use of weighted point sets, namely, sets of pairs of weight and vector,
as representations of instances. In this work, we theoretically show the benefit
of our proposed method through a new understanding of the contrastive loss of
CLIP, which we call symmetric InfoNCE. We clarify that the optimal similarity
that minimizes symmetric InfoNCE is the pointwise mutual information, and show
an upper bound of excess risk on downstream classification tasks of representations
that achieve the optimal similarity. In addition, we show that our proposed
similarity based on weighted point sets consistently achieves the optimal similarity.
To verify the effectiveness of our proposed method, we demonstrate pretraining of
text-image representation models and classification tasks on common benchmarks.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730574295211

thread_type:
official_review

<thread_object>
creator: reviewer_2vys

<content>
summary:
The paper proposes to replace the point-wise contrastive loss of CLIP with a dense contrastive loss calculated by using the contrastive loss on top of the average pair-wise similarity between dense features (the features prior to the final pooling layer of encoders). The authors show that this leads to better results (using the same evaluation as CLIP). The method is rigorously motivated by analyzing excess risk relative to the optimal downstream classifier head.

strengths:
(1) The introduction is especially well-written. I appreciated the motivation that many-to-many correspondences are not adequately represented by a single CLIP embedding.

(2) The theory is nice (but Eq. 3 has typos).

weaknesses:
(1) It is unclear how this method differs from dense contrastive loses. (For example: https://openaccess.thecvf.com/content/CVPR2021/papers/Wang_Dense_Contrastive_Learning_for_Self-Supervised_Visual_Pre-Training_CVPR_2021_paper.pdf)

(2) Computational cost. The contrastive loss is already quite computationally expensive (square of mini-batch size if not using global hard negative mining). Now we multiply that cost by the square of number of feature vectors. 

(3) Practically speaking (not considering the theory), there is not much innovation in this paper. Similarity metrics over pairs of sets of features are well-established. For example: ColBERT (https://arxiv.org/pdf/2004.12832). The extension from a pure NLP setting to image-text is straightforward.

questions:
None.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674910785

thread_type:
official_review

<thread_object>
creator: reviewer_ujlh

<content>
summary:
The authors presented an analysis of the contrastive loss of CLIP that gave insight on the optimal similarity measure for this loss. Furthermore, inspired by their analysis of the optimal loss, the authors proposed a new similarity measure that achives optimal similarity values, with reasonable computational cost.

strengths:
The paper presents an alternative to the inner-product similarity through the use of weighted sums of kernel function evaluations. As such, the method benefits from the kernel trick: whereas the simple inner-product similarity may require a high-dimensionality in feature space, the use of kernels reduces the dimensionality of the feature space and shortens the amount of calculations required. Conversely, the method allows for greater performance in downstream tasks for the same amount of computation, by approsimating more closely the (theoretical) optimal similarity measure.

weaknesses:
1. Inner-product similarity analysis

     A key point in the argument for the superiority of the authors' method over the standard similarity measure of CLIP is given in section 5.1, regarding the approximation error between the inner-product similarity matrix and the optimal similarity matrix $G$. There is an assumption that the number of samples $N$ is greater than the dimensionality $d$ of the feature space, thus the rank of the inner-product similarity matrix is smaller than the rank of $G$, implying the existence of approximation error. However, it is not clear from the paper that the approximation error is indeed deleterious, since the effective rank of $G$ could be substantially smaller than $N$ - and the effectiveness of CLIP suggests that this may indeed be the case. 
     Another evidence in this direction is the fact that the authors had to use a combination of a linear kernel with a nonlinear kernel in their approach, and their ablation study showed that the linear-only kernel has comparable performance (by exceeding in the zero-shot classification task, and underperforming in the linear classification task, and in both cases by a small margin that is not evaluated in regards to its statistical significance).

2. Need for linear-only and nonlinear-only results

     The hyperparameter options for $(\alpha_1, \alpha_2)$ do not include a linear only $(1, 0)$ and nonlinear-only $(0,1)$ option. Given the ablation results, such choices were desirable.

questions:
It would be interesting to have an analysis of:

- The effective rank of $G$ to clarify whether the stated deficiency of inner-product similarity is substantial. For instance, an analysis of the singular values of $G$ could be of value, since it would directly relate the rank disparity to the L2 error.

- The effectiveness of linear-only kernels in addition to the experiments already in place.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693883392

thread_type:
official_review

<thread_object>
creator: reviewer_sno6

<content>
summary:
The paper presents a new understanding of the symmetric contrastive loss InfoNCE of CLIP, and clarify the point-wise mutual information is the optimal similarity that minimizes symmetric InfoNCE., then gives an upper bound of excess risk on downstream classification tasks of representations when the optimal similarity is achieved. Then, the paper proposes similarity based on weighted point clouds to consistently achieves the point-wise mutual information and the corresponding implementation of the proposed similarity.

strengths:
1. The paper starts from a new understanding of contrastive loss, and proposes similarity measure that approximates the optimal formulation, which is theoretically well-founded.

2. The paper combines theory with practice and has demonstrated the effectiveness of the proposed similarity through experiments.

weaknesses:
1. The innovation is limited. Some core concepts of the paper, such as suboptimality decomposition, are referenced from existing work.

2. The role of weighted point cloud seems to be overstated. The paper did not use concepts related to point clouds, such as neighborhoods, connectivity. Only the “weight” is used in the weighted similarity aggregation.

3. Some concepts or expressions need further clarification. For example, 1. What exactly does the similarity structure (line 41) mean? 2. Why is the rank of G greater than d+1, there exists a certain error of the approximation of G? (lines 309-310) 3. The assumption C.2 seems strong, can you provide a specific explanation based on practical examples? 4. The equation 3 has empty before ||, which seems a typo.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732442155627

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Message to the Reviewers

comment:
Dear all reviewers,

We sincerely appreciate your taking the time to review our paper and are grateful for all the constructive and insightful feedback provided by the reviewers. In response to the feedback, we have made the following updates to our manuscript. The changes are highlighted in red in the updated version. We would appreciate it if you could take a look at it.
* We have corrected the typo in Theorem 4.2, as noted by Reviewers Sno6 and 2vyS.
* We have added explanations and results about ablation study, particularly regarding models that use only a non-linear kernel, as suggested by Reviewer UjLH. In conjunction with this additional experiment, we have expanded the hyperparameter search space for the coefficients $(\alpha_1, \alpha_2)$ and updated the corresponding tables.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work studies the pretraining and inference mechanisms of the alignment methods used in vision-language models (VLMs) such as CLIP. The authors provide new theoretical insights into the optimal form of embeddings to enhance downstream classification tasks and develop a kernel formulation based on these insights. Empirical evaluations demonstrate significant improvements over the vanilla form used in CLIP (based on cosine similarity). All reviewers acknowledge the novelty and significance of this study. The AC concurs with the reviewers' assessments and recommends the acceptance of this work, **congratulations**.  

Please address the following comments in the final version:  

- Explicitly explain the mechanics behind "[PAD] token being dependent on its relative position to [EOS]."  
- Revise Equation 3, as the use of the dot symbol (e.g., $.\mathcal{Y}_c$) is confusing.  
- Provide the parameters of the Random Fourier Features (RFF), specifically $\omega_t$ and $\beta_t$.  
- Explain why the dimensionality of the RFF differs between pretraining and zero-shot evaluations.
- In computer vision, point cloud refers to inferring 3D shape of an object or scene from raw 3D points. The usage of a point cloud in this context to refer to a set of fixed-sized to represent data might not be optimal. 
- The title of the paper may be revised. If insisting on the use of point cloud, a `:` should be placed after learning (Weighted Point Cloud Embedding for Multimodal Contrastive Learning: Toward an Optimal Similarity Metric)