Do I look like a `cat.n.01` to you? A Taxonomy Image Generation
  Benchmark
Authors: name: Viktor Moskvoretskii; name: Alina Lobanova; name: Ekaterina Neminova; name: Chris Biemann; name: Alexander Panchenko; name: Irina Nikishina
Published: 2025-03-13 13:37:54

Abstract: This paper explores the feasibility of using text-to-image models in a
zero-shot setup to generate images for taxonomy concepts. While text-based
methods for taxonomy enrichment are well-established, the potential of the
visual dimension remains unexplored. To address this, we propose a
comprehensive benchmark for Taxonomy Image Generation that assesses models'
abilities to understand taxonomy concepts and generate relevant, high-quality
images. The benchmark includes common-sense and randomly sampled WordNet
concepts, alongside the LLM generated predictions. The 12 models are evaluated
using 9 novel taxonomy-related text-to-image metrics and human feedback.
Moreover, we pioneer the use of pairwise evaluation with GPT-4 feedback for
image generation. Experimental results show that the ranking of models differs
significantly from standard T2I tasks. Playground-v2 and FLUX consistently
outperform across metrics and subsets and the retrieval-based approach performs
poorly. These findings highlight the potential for automating the curation of
structured data resources.