Leveraging Knowledge Graphs to harvest a high-quality dataset for efficient CLIP model training

Authors: Simon Ging, Sebastian Walter, Hannah Bast, Thomas Brox
Primary area: datasets and benchmarks
Keywords: vision-language, image-text, contrastive learning, CLIP, dataset, knowledge graph, wordnet, wikidata, entities, attributes, image search, imagenet, inaturalist, cub, ovad, attribute classification, animals, plants, small data, efficiency, open vocabulary

Abstract:
Vision-language contrastive learning based on the CLIP method has been instrumental in driving recent advancements in computer vision. However, high quality CLIP models are based on very large datasets. This makes them expensive to train and hampers the scientific analysis of these models. We show how to train a CLIP base-size model efficiently for a broad domain on a much smaller amount of data. We demonstrate this specifically with the automated creation of a dataset named LivingThings with 8.9M images of animals and plants and 12.2M texts. The dataset is obtained via focused image-search queries of three kinds: entity queries (e.g., "eagle"), entity-attribute queries (e.g., "bushy tail of a fox"), and type-attribute queries (e.g., "insect on a leaf"). The entities and types, as well as some of the texts, are derived from the WordNet and Wikidata knowledge graphs, the attributes are obtained via LLMs. We train a CLIP model from scratch on LivingThings and evaluate it on ImageNet, iNaturalist, and CUB for object classification and OVAD and CUB for attribute classification. On the broad target domain of animals and plants, our model achieves comparable, and sometimes even much better performance than models that have orders of magnitude more parameters or training data. For instance, our ViT-B-32 model improves over much larger state-of-the-art CLIP models on the iNaturalist 21 object classification task. We will publicly release our code and dataset.

Decision: reject
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730426685900

thread_type:
official_review

<thread_object>
creator: reviewer_eled

<content>
summary:
This paper proposes a new dataset, named LiveThings, with 8.9M images and 12.2M text scale, leveraging the knowledge graph, Large Language Model, and image search engines. Specifically, the knowledge graphs (e.g., WordNet, Wikidata) output the entity classes and natural types, following the generation of attributes via LLMs. Following the image searching and filtering process, the author proposes a high-quality datasets for animal and plant, which is for biological domain.

strengths:
1. The author proposes a new dataset, LiveThings, to efficiently train the CLIP-based model, which could be leveraged for future works involving foundation model construction.
2. Through extensive experiments, the paper demonstrates the superiority of utilizing automatically generated datasets, especially in efficiency.
3. The paper is well-written, allowing readers to understand easily.

weaknesses:
1. It seems necessary to consider the additional baseline [1]. The proposed dataset, which focuses on biological information containing animal and plant entities, shares similarities in size with another dataset called TreeOfLife [1] that also contains biological data and spans 10M sizes. Therefore, to better demonstrate the advantage of using a knowledge graph in developing their dataset, the authors should perform a comparative analysis with a model that has been trained on the TreeOfLife dataset.

Since it was trained in the biological domain, such as animals and plants, with attribute information, it is natural that it performs well in attribute classification and species classification while showing low performance on other domains (e.g., other in ImageNet). Therefore, it seems necessary to compare it with models trained in the biology domain using a dataset of a similar scale (i.e., TreeOfLife) to validate the effectiveness of leveraging knowledge graphs.

2. There is no experiment for other domains to support the claim that this approach can be used for any domain covered by the given knowledge graph. In the whole paper, only the biological domain is covered, but it is crucial that this approach indeed can cover other domains, such as urban and molecular. The reason for this doubt is that the entities in the knowledge graph may not be diverse in other domains, or it might not be effective in extracting natural types. Therefore, it seems necessary to conduct experiments to prove this.

[1] BioCLIP: A Vision Foundation Model for the Tree of Life. Stevens et al. CVPR'24

questions:
The goal of this paper is to create high-quality data using the knowledge graph, but it seems to be utilized only for extracting entity classes or natural types. To fully leverage the knowledge graph, it appears necessary to use the structural information about the entities within it. Extracting entity-related attributes from the knowledge graph is more natural than using an LLM. 
Regarding it, what if the attributes are extracted based on the structural information within the knowledge graph instead of using LLMs? I think it is more natural to fully utilize the knowledge graph's information.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730522957201

thread_type:
official_review

<thread_object>
creator: reviewer_njby

<content>
summary:
This paper proposes to harvest small-scale but high-quality data for CLIP training leveraging existing knowledge graphs. For a proof of concept, the paper focuses on visible living things including animals and plants. By extracting entities from knowledge graph, generating attributes and building queries using LLM, and collecting images using Google API, the authors created a dataset named LivingThings with 8.9M images and 12.2M texts. With the collected dataset, the authors trained a CLIP ViT-Base model from scratch and achieve comparable or better performance over large-scale trained CLIP variants.

strengths:
1. The paper is well organised with clear motivation. The overall flow is easy to follow.

2. The work contributes a high-quality dataset named LivingThings covering animal and plant entities, which I believe would benefit the community.

3. This work is quite resource-efficient, training a CLIP-style model for visual recognition using only 8 RTX 2080 Ti GPUs. And each round of training only costs ~30 hours.

weaknesses:
1. The model studied in this work is ViT-Base, a small variant of the CLIP family. Therefore, the scalability of model size using the LivingThings dataset might need further verification.

2. The method relies on the prior knowledge of the root entity (e.g., plant and animal) when collecting the dataset. Therefore, the proposed paradigm seems only suitable for visual recognition in a single or limited number of domains.

questions:
1. LivingThings has a total of 8M images. I am curious about the model performance of using even fewer images, e.g., 100K, or 1M.

2. How would the proposed paradigm be extended to build a CLIP-level model that covers unlimited domains? And how many images might be needed to match the performance of standard CLIP models? Just some analyses would suffice.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697706613

thread_type:
official_review

<thread_object>
creator: reviewer_rb9j

<content>
summary:
This paper tries to propose a controlled-generated image-pair dataset, where the query entity is from the Wikidata knowledge graph, and the attribute and search query are generated using LLMs. Then, search engines are used to collect the images. Later, it proves that the 8.9M dataset can train the model with comparable performance of larger dataset on living things categories.

strengths:
1.	The goal is to investigate whether a high-quality small-scale dataset can achieve comparable performance to the CLIP model trained on large data.
2.	Create the datasets that are based on the Wikidata knowledge graph, ensuring wide data coverage over different categories.
3.	The query creation + searching is a good way to construct high-quality dataset.

weaknesses:
1.	This paper collected the LivingThings dataset covering animals and plants, but it is still small scale and the model trained from it cannot compare with SOTA CLIP model. I wonder if anything this paper can do to further improve the SOTA CLIP model by using this datasets? 
2.	This dataset only has a local coverage of animals and plants. But I wonder how can this data collection pipeline be further scaled up to the next DataComp or LAION sized dataset. If the author keeps increasing the entity number from wikidata, it might be piled up by including other categories, and finally still result in a very large dataset in billion scale, then you will have to prove it is better than LAION or DataComp
3.	I would expect more detailed experiment analysis and ablation study about the effect of attribute number, number of entities, scaling law, etc.
4.	The quality control of data collection seems to be loose. Seems it cannot guarantee the alignment of text and image is high. Why not use a CLIP to filter out the unaligned image-text pair?

questions:
I think the main concern is that the dataset itself is less in data analysis and experimental ablation. Also, it cannot surpass the SOTA CLIP, and didn't show clearly how it can help boost SOTA CLIP.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732139430770

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their insightful comments and questions and for taking time to review our work. We have uploaded a new submission and added new content to the appendix beginning on page 24. Notably, we compared our approach to BioCLIP and compared all models on the Rare Species dataset, created a study on dataset scale, and described how we can extend our data domain from animal and plant data to the entire visual world. We also compare pretraining and finetuning on our dataset. All new content is indicated in blue, for new sections we mark the section title, and for new figures and tables we mark the caption.

In the interest of responding to the reviews now, and due to the page limit, we have postponed integrating the new content into the main paper and have added everything to the appendix. We will integrate parts of the appendix content into the main paper until the end of the discussion period, notably we will update the main paper with the results of BioCLIP, finetuning, and the Rare Species dataset.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposed to leverage knowledge graph as the guidance to collect high-quality data for CLIP model training, particularly focused on animal and plant categories. The authors start with a knowledge graph and further extend the query by adding attributes and types using LLMs. The resulted dataset called LivingThings consists of 8.9M images and 12.2M texts. Afterwards, the authors trained a CLIP model from scrach on the dataset and evaluate it on various benchmarks including ImageNet, iNaturalist and CUB for image classification. The trained model showed superior performance on these benchmarks and even outperforms those trained with much larger dataset and more parameters.

The main strength of this work is introducing a high-quality dataset LivingThings which is the first ever vision-language dataset that are specifically designed for animals and plants. Based on this dataset, the authors pretrained a CLIP model from scratch and demonstrate good performance acorss various benchmarks.

As pointed out by the reviewers, however, the proposed method of constructing image-text data for CLIP training is not necessarily generalized to new domains. Despite some efforts to prove this during the rebuttal, it still cannot resolve fully the reviewers' concern. In additiona, the diversity of the collected data is also concerned as all the queries are generated based on the entities in the knowledge graph. At last, the ACs also think this work does not show the generality of the proposed method if the goal is to leverage knowldege graph to harvest a high-quality dataset for CLIP training, regardless of the domain of interests.

After the rebuttal session, this work got 5,5,6 from three reviewers. Based on the reviews and discussions, the ACs think this work still has some obvious flaws, as thus suggest it is not ready for this venue.