ICML 2025

Bongard in Wonderland:
Visual Puzzles that Still Make AI Go Mad?
Antonia W¨ust 1 Tim Woydt 1 Lukas Helff 1 2 Inga Ibs 3 4 Wolfgang Stammer 1 2 Devendra S. Dhami 5
Constantin A. Rothkopf 2 3 4 Kristian Kersting 1 2 4 6
Abstract
Recently, newly developed Vision-Language
Models (VLMs), such as OpenAI’s o1, have
emerged, seemingly demonstrating advanced rea-
soning capabilities across text and image modal-
ities.
However, the depth of these advances
in language-guided perception and abstract rea-
soning remains underexplored, and it is unclear
whether these models can truly live up to their
ambitious promises. To assess the progress and
identify shortcomings, we enter the wonderland
of Bongard problems, a set of classic visual rea-
soning puzzles that require human-like abilities of
pattern recognition and abstract reasoning. With
our extensive evaluation setup, we show that while
VLMs occasionally succeed in identifying dis-
criminative concepts and solving some of the
problems, they frequently falter. Surprisingly,
even elementary concepts that may seem trivial to
humans, such as simple spirals, pose significant
challenges. Moreover, when explicitly asked to
recognize ground truth concepts, they continue to
falter, suggesting not only a lack of understanding
of these elementary visual concepts but also an
inability to generalize to unseen concepts. We
compare the results of VLMs to human perfor-
mance and observe that a significant gap remains
between human visual reasoning capabilities and
machine cognition. 1
1AIML Lab, TU Darmstadt 2Hessian Center for AI (hessian.ai)
3Institute of Psychology, TU Darmstadt 4Centre for Cognitive
Science, TU Darmstadt 5Uncertainty in AI Group, TU Eindhoven
6German Center for AI (DFKI). Correspondence to: Antonia W¨ust
<antonia.wuest@cs.tu-darmstadt.de>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1Our
code
is
available
at
https://github.com/...