Compositional Entailment Learning for Hyperbolic Vision-Language Models

Authors: Avik Pal, Max van Spengler, Guido Maria D'Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, Pascal Mettes
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Vision-Language Models, Hyperbolic Geometry, Representation Learning, CLIP

Abstract:
Image-text representation learning forms a cornerstone in vision-language models, where pairs of images and textual descriptions are contrastively aligned in a shared embedding space. Since visual and textual concepts are naturally hierarchical, recent work has shown that hyperbolic space can serve as a high-potential manifold to learn vision-language representation with strong downstream performance. In this work, for the first time we show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-text pairs. We propose Compositional Entailment Learning for hyperbolic vision-language models. The idea is that an image is not only described by a sentence but is itself a composition of multiple object boxes, each with their own textual description. Such information can be obtained freely by extracting nouns from sentences and using openly available localized grounding models. We show how to hierarchically organize images, image boxes, and their textual descriptions through contrastive and entailment-based objectives. Empirical evaluation on a hyperbolic vision-language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP learning, as well as recent hyperbolic alternatives, with better zero-shot and retrieval generalization and clearly stronger hierarchical performance.

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730548706372

thread_type:
official_review

<thread_object>
creator: reviewer_5zgk

<content>
summary:
This paper proposes the novel Compositional Entailment Learning framework to train VLMs, by using as supervision the hierarchical relations between images, captions, and constituent nouns and their bounding boxes. Their results show that this outperforms standard CLIP and the hyperbolic CLIP variant MERU on both standard multimodal and hierarchical benchmarks. This is supported by qualitative results illustrating the learned hierarchical semantics of the learned space.

strengths:
The central idea is clever and novel – utilizing the hierarchical nature of nouns mentioned in image captions as supervision for a hyperbolic model. The exposition is clear and concepts are well-illustrated. The quantitative experiments are extensive and overall convincing.

weaknesses:
Qualitative results (Sec 4, Supp 8) are fairly limited. In particular, it is missing a qualitative comparison to existing models (CLIP, MERU) to illustrate whether HyCoCLIP’s embedding space represents hierarchies in a more qualitatively satisfying way.

While a comparison to CLIP trained from scratch is provided, recent work has found pretrained foundation VLMs to represent hierarchies in Euclidean space [1]. It would be useful to compare to such results to understand whether HyCoCLIP trained from scratch is competitive with such models.

[1] Alper and Averbuch-Elor. Emergent Visual-Semantic Hierarchies in Image-Text Representations. ECCV 2024

questions:
Could the use of objects as supervision bias the model towards nouns and concrete concepts, possibly at the expense of attributes, dynamic actions (verbs), etc.?

Some details that are unclear from Supp. A: How were abstract nouns filtered? Are the nouns that can be grounded open-vocabulary (not limited to a fixed list)? How accurate is the GLIP-based grounding procedure?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688699918

thread_type:
official_review

<thread_object>
creator: reviewer_uef2

<content>
summary:
The authors proposed to incorporate hierarchical pretraining for hyperbolic vision language models and the resulting model Hyperbolic Compositional CLIP (HyCoCLIP). The core idea is to construct object regions (image boxes) and corresponding text phrases (text boxes) to build a multi-layered, compositional hierarchy within the shared hyperbolic embedding space. The HyCoCLIP shows competitive performance in zero-shot classification and retrievals. The author also conducted experiments to show how HyCOCLIP can outperform CLIP and the hyperbolic contrastive model MERU in zero-shot hierarchical classification and scene understanding tasks.

strengths:
I think this paper is very well written and I find it easy to follow. Overall the idea behind HyCoCLIp is well motivated and I believe the authors have conducted sufficient experiments to empirically demonstrate the proposed method and model’s efficacy. The empirical performance of HyCoCLIP is very strong and to the best of my knowledge, the proposed HyCoCLIP achieved the state-of-results on many of the reported zero-shot tasks from a contrastive-pretrained model.

weaknesses:
One major concern is the incremental nature of this work. Hyperbolic embeddings for representing hierarchical relationships have been explored in previous models, and this paper primarily builds upon these established ideas. However, the specific contributions of HyCoCLIP, particularly in enhancing hierarchical and scene understanding tasks, offer sufficient merit to make this work valuable to the broader community.

questions:
In Table 1/2, the authors bold the best performance overall across different model backbones. Wouldn’t it be more informative and fair to bold the best performance within each backbone group (e.g., ViT-S/16, ViT-B/16) to allow for a clearer comparison of HyCoCLIP’s performance relative to baselines on similar architectures?

Regarding the choice of batch size, the authors used a batch size of 768 due to memory limitations. Did the authors consider implementing techniques like gradient accumulation to effectively simulate a larger batch size? This could provide further insights into how batch size impacts model performance, especially since batch size has been shown to affect contrastive learning tasks significantly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730938528823

thread_type:
official_review

<thread_object>
creator: reviewer_cujb

<content>
summary:
This paper introduces a novel approach named HyCoCLIP to vision-language modeling that leverages the hierarchical nature of hyperbolic space to better align visual and textual data. It organizes image and text data as hierarchical compositions, where objects within an image and their corresponding text descriptions are represented at different levels of abstraction in hyperbolic space.The experiments demonstrate that HyCoCLIP achieves significant performance improvements across multiple tasks.

strengths:
1. This paper is well-organized. The motivation is easy to follow, and the method is easy-to-understand.
2. The proposed HyCoCLIP is novel and effective. It organizes data at multiple abstraction levels, providing an inspiring approach to multi-modal learning.
3. The authors performs exhaustive experiments to show that the effectiveness of HyCoCLIP. It outperforms baselines on general and fine-grained image classification tasks.

weaknesses:
1. While the paper compare with CLIP and MERU, it should also compare some recently proposed VLMs.
2. The paper should explore how sensitive the model is to the choice of hyperbolic space parameters.

questions:
Could you please provide more details on the choice of hyperbolic space parameters?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730961799243

thread_type:
official_review

<thread_object>
creator: reviewer_spcr

<content>
summary:
This work proposes a novel learning method for training vision-language models. Specifically, the method involves pretraining such models with 2 losses --- hierarchical compositional contrastive and entailment losses. The hierarchical concepts correspond to image boxes and the corresponding text boxes. The experiments are conducted on large scale dataset (GRIT) consisting of 20.5M image-text pairs. In Appendix A, the authors describe an automatic procedure to obtain the text boxes (noun entities in this case) and their corresponding bounding boxes in the images. The paper details empirical results on a variety of tasks including zero-shot image classification, retrieval, object detection and scene understanding.

strengths:
* The proposed method is simple and elegant and can be easily applied to large scale pretraining of vision-language models. The procedure to automatically generate paired image and text boxes is also relatively straightforward.
* The empirical results show improvement across several tasks which demonstrates the improved representation learning - classification, retrieval, detection and understanding.
* Table 1 results show that CLIP trained on additional image-text boxes doesn't improve the performance. However, training on the same data but with the proposed hierarchical compositional learning losses shows significant improvement in performance. This further demonstrates the effectiveness of the proposed technique.

weaknesses:
When training CLIP on additional image-text boxes shows no improvement (Table 1), it could be because there is limited new information in such examples (as original image-text pairs are already present in the training data). For a better understanding of this, an experiment such as this might help: split the GRIT dataset into 2 random subsets of 10M each. Then compare the results on the following settings:

[1] CLIP trained on 10M image-text pairs

[2] CLIP trained on 10M image-text pairs + additional image-text boxes

[3] HyCoCLIP trained on 10M image-text pairs + additional image-text boxes

[4] CLIP trained on 20M image-text pairs

The paper presents the comparison of [1] vs [2] vs [3] (but on all 20M image-text pairs) in Table 1 but comparing [3] vs [4] will help answer the above question. It is worth noting that even if the comparison shows similar results, [3] might still be slightly favored since it can be applied on top of any existing large dataset.

questions:
Can the authors share the results of HyCoCLIP on RedCaps dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732366024938

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their valuable feedback and for positively rating our work. We appreciated the shared comments about the clarity of the writing and the novelty of our proposed approach. In the revised manuscript, we implemented suggestions and added several details of HyCoCLIP that were recommended by the reviewers, which we think contribute to strengthening our work. These additions can be found in the revised manuscript, highlighted in red.

We have included several comprehensive qualitative results in Appendix I. Additionally, including illustrations from the GRIT dataset eases the understanding of the grounding procedure of Appendix A while also providing qualitative insights. We have also revised Table 1 following the reviewers' suggestions and report the sensitivity of HyCoCLIP to the curvature of the hyperbolic space in Table 7 of Appendix B. Additionally, the results of the zero-shot evaluation of HyCoCLIP on the HierarCaps dataset are in Table 10 of Appendix H.

In the following, we address the questions and weaknesses raised by the reviewers to the best of our understanding and remain open to additional feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the hierarchical visual-text representation. Concretely, the author proposed to leverage the hierarchical relation within the image (whole image and objects) and the text (whole sentence and nouns) to construct a hierachical embedding space, where the more general terms (objects / nouns) are pushing towards the origin and the more specific terms (sentences and whole images) are pushing towards the boundary. To construct this hierarchical embedding space, the author proposed object (noun) and sentence (image) contrastive loss and also entailment loss. 

Strength:
1. The paper is easy to follow and well-written
2. The proposed approach is interesting and novel.
3. The proposed approach achieves good performance on multiple benchmarks.

Weakness:
1. Considering the related works on hierarchical and hyperbolic space, the proposed approach might be slightly incremental. 

Given all the support from the reviewers (all 8), I recommend accept.