Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World
Authors: Rujie Wu
Xiaojian Ma
Zhenliang Zhang
Wei Wang
Qing Li
Song-Chun Zhu
Yizhou Wang
Venue: ICLR 2024 poster (2024)
Keywords: Few-shot learning
Visual reasoning
Open world learning
Primary area: datasets and benchmarks

Abstract: We introduce Bongard-OpenWorld, a new benchmark for evaluating real-world few-shot reasoning for machine vision. It originates from the classical Bongard Problems (BPs): Given two sets of images (positive and negative), the model needs to identify the set that query images belong to by inducing the visual concepts, which is exclusively depicted by images from the positive set. Our benchmark inherits the few-shot concept induction of the original BPs while adding the two novel layers of challenge: 1) open-world free-form concepts, as the visual concepts in Bongard-OpenWorld are unique compositions of terms from an open vocabulary, ranging from object categories to abstract visual attributes and commonsense factual knowledge; 2)  real-world images, as opposed to the synthetic diagrams used by many counterparts. In our exploration, Bongard-OpenWorld already imposes a significant challenge to current few-shot reasoning algorithms. We further investigate to which extent the recently introduced Large Language Models (LLMs) and Vision-Language Models (VLMs) can solve our task, by directly probing VLMs, and combining VLMs and LLMs in an interactive reasoning scheme. We even conceived a neuro-symbolic reasoning approach that reconciles LLMs & VLMs with logical reasoning to emulate the human problem-solving process for Bongard Problems. However, none of these approaches manage to close the human-machine gap, as the best learner achieves 64% accuracy while human participants easily reach 91%. We hope Bongard-OpenWorld can help us better understand the limitations of current visual intelligence and facilitate future research on visual agents with stronger few-shot visual reasoning capabilities.

Published as a conference paper at ICLR 2024
BONGARD-OPENWORDD. FEW-SHOT REASONING FOR FREE-FORM VISUAL CON- CEPTS IN THE REAL WORLD
Rujic Wu*1 Xiaojian Ma Zhenliang Zhang²
Wei Wang X 2
Qing Li Lix Song-Chun 2hu Yizhou Wang
School of Computer Science, Peking University
"National Key Laboratory of General Artificial Intelligence, BIGAI
"School of Intelligence Science and Technology, Peking University
*Institute for Artificial Intelligence, Peking University
* Equal contribution
X Co-corresponding authors
Project page: Bongard-Open World Worlll
ABSTRACT
We introduce Bongard-Open few-shot reasoning for machine vision. It originates from the classical Bongard Problems (BPs): Given two sets of images (positive and negative), the model needs to identify the set that query images belong to by inducing the visual concepts, which is exclusively depicted by images from the positive set. Our benchmark inherits the few-shot concept induction of the original BPs while adding the two novel layers of challenge: 1) open- world free-form concepts, the visual concepts in Bongard-Open World are unique compositions of terms from an open vocabulary, ranging from object categories to abstract visual attributes and commonsense factual knowledge; 2) real-world images, as opposed to the synthetic diagrams used by many counterparts. In our exploration, Bongard-Ope World already imposes a significant challenge to current few-shot reasoning algorithms. We further investigate to which extent the recently introduced Large Language Models (LLMs) and Vision-Languag, Models (VLMs) can solve our task, by directly probing VLMs, and combining VLMs and LLMs in an interactive reasoning scheme. We even conceived neuro-symbolic reasoning approach that reconciles LLMs & VLMs with logical reasoning to emulate the human problem-solving process for Bongard Problems. However, none of these approaches manage close the human- machine gap, as the best learner achieves 64% accuracy while human participants easily reach 91 %. We hope Bongard-Open World can help better understand the limitations of current visual intelligence and facilitate future research on visual agents with stronger few-shot visual reasoning capabilities.
new benchmark for evaluating real-world
INTRODUCTION
In recent years, substantial progress has been recorded in developing visual intelligence. Given an image, visual agents now can robustly recognize the presenting objects (object detection and segmentation (Deng et al., 2009; He et al., 2017; Lin et al., 2014)), describe what's happening (image captioning (Xu et al., 2015; Li al., 2023)), and even answering complex questions about it (visual question answering (Goyal et al., 2017; Hudson & Manning, 2019; Ma et al., 2022a)). However, completing many of these tasks requires a massive amount of training data. On the contrary, humans can perform more sophisticated tasks with very few visual inputs (Marr, 2010; Zhu & Zhu, 2021; Huang, 2021). For example, humans can recognize and reason about compositional real-world visual concepts from just few examples (Lake et al., 2015; Zhang et al., 2019; Nie et al., 2020; Jiang et al., 2022), i.e. few-shot visual reasoning. To facilitate human-level visual intelligence, it is necessary to go beyond canonical tasks and develop new benchmarks that aim to comprehensively evaluate few-shot learning of novel and complicated visual concepts.
Several benchmarks have been introduced with focus few-shot learning of visual concepts, such as Omniglot (Lake et al., 2015), minilmageNet (Vinyals et al., 2016), and al., 2020). However, these datasets primarily focus identifying simple object categories instead
(Triantafillou

---

Published as a conference paper at ICLR 2024
query image l
Two zebras are ranning in the
A bear ranning with many hirds Biiing acound it
A dog is running on the besch.
positive set p
heed wäldebesst are running in graollndd
herd schras are drinking the river.
A host is catching the shallow river.
A humanoid robot is manning
l belongs to P or N ? (optional) What is the concept exclusively depicted by P?
negative set NN
Ground fruth conceept (sw sentenee):
antimiis are rwvniing
Figure 1: Task illustration of Bongard-Open World. Given two set of images P and V, the model needs to identify which set the query image I, belongs to by inferring the concepts C that is exclusively depicted by P. Note that the captions and the concepts C won't be provided to the model. To further increase the difficulty of our task, we introduce distractors as additional contents the positive images other than the concept C, and hard negatives to ensure the content of negative images partially overlaps with the concepts C. These practices could force the model to reason about the visual concepts by contrasting the positives and the negatives.
of novel and complex visual concepts, e.g. compositions of objects and their characteristics (Krishna et al., 2017) and visual relationships (Chao et al., 2015), and generalization of attributes (Ren et al., 2020). Many benchmarks have been dedicated to abstract visual reasoning and are considered to be few-shot learning, such as RPM (Raven-style Progressive Matrices) (Zhang et al., 2019; Barrett et al., 2018) and Bongard Problems (Nie et al., 2020). While offering interesting visual tasks, these benchmarks are prone to use synthetic graphics instead of real-world images and stay with basic object-level geometrical concepts such as shapes, sizes, amounts, etc. The closest benchmark to work is Bongard-HOO HOI (Jiang et al., 2022), which features few-shot visual reasoning and compositional visual concepts (human-object interactions, i.e. HOI) in the real world. But since it is built upon an existing close- vocabulary image recognition dataset (HAKE (Li et al., 2019)). the total number of unique concepts (242, while Bongard-Open World has 1.01k) can be limited. Moreover, the concepts are restricted to be HOIs with a fixed structure (object, interaction), which deviates from the free-form nature of visual concepts in the real world.
To solve the aforementioned issues and step towards more general few-shot visual reasoning challenge that invites all possible contestants, we introduce Bongard-OpenWorrd, a benchmark that reconciles the best of all parties: few-shot learning, real-world images, and compositional visual concepts. Longard-OpenWeee World inherits the simplicity of the original Bongard Problems: given six positive and six negative images, along with two query images, the goal is to identify the visual concepts, which is exclusively depicted by the positive images, and make binary predictions on the query images (see Figure I for illustrations, their quantity here is halved to facilitate a clearer visualization of image details).
At the heart of our benchmark open vocabulary free-form concepts and illustrated in Table 1. Instead of using a small and pre-defined set of concepts that are predominately about the same topic (object attributes in Bongard-LOGO (Nie et al., 2020), HOIs in Bongard-HOI (Jiang al., 2022), etc.), we leverage Conceptual Captions (CC-3M) (Sharma et al., 2018), a massive web-crawled collection of rich and open vocabulary image descriptions. We further propose grid sampling to extract visual concepts from the streamlined captions. The resulting visual concepts are therefore both free-form, with no assumption on the structure, e.g. 2-tuple as in Bongard-HOII and open vocabulary. We also crowd-source some challenging concepts, including abstract visual attributes, and commonsense factual knowledge (examples can found in Table 1), and compose them with the concepts extracted from CC-3M. Finally, we manually verify the plausibility of our generated visual concepts. We end up with 1.01K unique concepts, and 26.6% of them are crowd-sourced challenging concepts. Statistics of these visual concepts can be found in Figure 2.
We then construct Bongard-Opent World problems out of the aforementioned visual concepts. Each problem is assigned a positive concept and online image search tool is used to find the most relevant images. Additionally, we follow the practice in (Jiang et al., 2022) to introduce distractors and hard negatives. We collect positive images containing additional content distractions other than the visual concepts C. Moreover, by partially modifying the positive concept to produce negative concepts, we ensure the content of the negative images partially overlaps with the positives. Therefore,
2

---

Published as a conference paper at ICLR 2024
Table I: A catalog of visual concepts in Bongard-Open We demonstrate the categories of concepts mined from CC-3M, and challenging commonsense-rllatee concepts that srouddsoureed and augmented to the dataset, "denotes gommongense. More examples see Table
Concept Caregory
ID
Example
Concept Calegrey
ID
Example
Anything clse"
Animals are running.
And / Or Not
5
A man without heard.
HOI Nutrition/
person playing guitar- plate of high-cakorie
Factual Knowledge
A building in US capital.
Class
Pelidae animals.
Coloe/ / Material Shape
A wooden floor in living room. An animal eapabli of nlinn in the tree.
Relationhip
A bench pear trees. Refraction of light on a glass cup.
Punctionality/ / Stanis/ Affoniance
Unusual Obscrvations
reasoning about the visual concepts by contrasting the positive and negative images will be required to solve the problem, rather than merely recognizing some simple common contents among the positive images (illustrated in Figure 1). Overall, we produce 1.01 high-quality Bongard-Open World WorlldldpWWdddd
problems. Thanks to the diverse set of concepts, each problem has its unique visual concepts. Comparisons with counterpart benchmarks can be found in Table 2.
a) Concept Length Distribution
b) Concept Category Distribution
c) Word Distribution
10.2%
65
0.006
0.007
ing
Lowe wate
0.005
4%
0.004
0.003
73 4%
29.1% 1%
0.002
0.000
Figure 2: Statistics Songard-Open World. Our benchmark exhibits a range of concept lengths, spanning from 2 to 5 (as depicted in subfigure a). with an average length of 3.3. As demonstrated in Table I, crowd-sourced commonsense concepts take ID 1-9, with 0 indicating "anything else" (as depicted in subfigure b). While some words are more frequent (sce the word cloud, as depieted in subfigure c), the overall frequency of words in Bongard-Open World concepts follows long-tailed distribution.
Benchmarking existing few-shot reasoners: In our experiments, we examine state-of-the-arr few- shot learning approaches, including non-episodic, meta-learning, and transformer-bseee models. Although the best few-shot learner (SNAIL (Mishra al., 2018)) shows some promising results on World with 64% average accuracy (the chance performance is 50%), the overall gap to human performances (91% of human contestants) is still substantial. To understand the failure mode, we hypothesize that visual pretraining could be pivotal due to the open vocabulary nature of our task. Therefore we investigate combining the learner with several pretrained visual representations. The results confirm that few-shot learners fueled with proper open-ended pretrained models, e.g. CLIP (Radford et al., 2021) can alleviate this gap. Further, we investigate to which extent the recently introduced Vision-Language Models (VLMs) and Large Language Models (LLMs) can approach Bongard-Oper Saaraadd-ponWolleee World. by directly probing VLMs, and combining VLMs and LLMs in an interactive reasoning scheme (Zhu al., 2023; Gao et al., 2022). We also developed neuro-symbolic reasoning method that integrates LLMs and VLMs with logical reasoning, it aims to mimic human problem-solving processes in addressing Bongard Problems. Our results indicate that despite the impressive results these approaches have attained in other tasks, they can still be confused by similar content between positive and negative examples in Longard-Open World World and therefore fail to close the human- machine gap.
Our contributions are summarized follows:
We introduce Bongard-Opent OorrdddddpWdd,,, new benchmark for few-shot visual reasoning with visual concepts in the real world, aiming at reconciling the challenging capabilities of few-shot learn- ing and reasoning with abstract and complicated real-world visual concepts and facilitating the development of human-like visual intelligence.
We carefully curate Bongard-Open World to include open vocabulary and free-form visual concepts, ranging from object categories to abstract visual attributes and commonsense factual knowledge. We also use distractors and hard negatives to make merely recognizing some common contents in the positives insufficient to complete our tasks.
We conduct extensive analysis on state-of-the-an few-shot reasoners including canonical few- shot learning systems and powerful LLMs, VLMs, and even neuro-symbolic learners. However, empirical results indicate that these approaches are struggling to match human performances on Bongard- Open Our findings suggest that robustly capturing sophisticated (compositional, abstract, factual or commonsense, etc.) visual concepts across multiple visual stimuli can still be a huge challenge even today's 's vision models.
3

---

Published as a conference paper at ICLR 2024
Table 2: An overview of benchmarks covering free-form image concepts, few-shot learning, and visual reasoning. The abbreviation vocab. denotes vocabulary and afti. denotes attributes. "We consider benchmark be open vocabulary when it collected without assuming a fixed set of visual concepts (e.g. CC-3M (Sharma et al., 2018),, or the assumed set is substantially large (e.g. Deta-Dataset (Triantafillou et al., 2020)). TTThe object attributes and object counts can he freely composed in V-PROM (Teney et al., 2020) but the object (0)
and interaction (I) in Bongard-HOO (Fiang et al., 2022) cannot.
datasel
concept
free-form concept
open vocab."
real-world images
low- shot
hard negatives
#concepts
Aasks
CC-3M (Sharna ot al,, 2018)
image caption
/
/
x
3L.IK
3.3M
Omniglot (Lake 2015)
shape image label
x
50
162K
minillageeNet (Vinyals et al., 2016)
x
x
/
/
X
100
60K
Meta-Datasct (Triantafillou et 2020)
imago label
/
/
4,934
52.8M
RPM (Barrett et al., 2018)
shape shape aitr, & count
X
X
/
50
11,36M
Rongard-LOOiD (Nie et al., 2020)
X
627
12K
V-PROM (Teney et 2020)
2020)
500
x
X
478
235K
Bongard-HOO (Jiang et al., 2022)
HOI
53K
Songard-OpenWortde (ours)
image caption
5
1001
L01K
2 THE BONGARD-OPEN
BENCHMARK
A problem instance in Bongard- World World can be formulated as a tuple (C. P,,, Iq),, where denotes the free-form compositional visual concepts, e.g. animals are running in Figure 1. The C is exclusively depicted by the positive images P. while all images from the negative set NN' do not depict it but might partially contain some of its terms c. The task image Lg: to determine whether C' can be found in L or not. While it is optional, the learner could explicitly induce the visual concepts C via captioning to offer some explainability of its prediction on lq. The following sections will detail how to collect the visual concepts and build the benchmark.
make binary prediction
the query
2.1 COLLECTING FREE-FORM OPEN VOCABULARY VISUAL CONCEPTS
Visual concepts C in Songard-OpenWonn visual concepts C. As anticipate it to be free-form and open vocabulary in our benchmark, C is effectively an arbitrary sentence that describes the content depicted by all images from the positive set P exclusively. However, since not all words in the sentence are meaningful to its semantics, we may streamline and convert it into a tuple of words and define the length of visual concepts C as the number of words in the tuple. Longer C will be more difficult be identified.
We begin with a more detailed illustration of the
Grid-sampling of visual concepts from CC-3M. We propose to extract these visual concepts from CC-3M (Sharma et al., 2018), a massive dataset of image-text pairs with comprehensive collection of open vocabulary free-form image descriptions. We then streamline all the captions into concepts tuples and perform grid sampling. Due space limitations. we only provide its key operations here and more details and pseudo-code can be found in the Appendix C: 1) We run sliding window with size 2/3/4/5 over all the concepts tuples to count the frequency of concepts with length 2/3/4/5: 2) Instead of sampling from the whole CC-3M "concept tuples" (refer to candidates for the visual concepts, where each concept conprises a tuple of words, ex. <red, tie>), we first split it into seveal small pools, or as we name it, "grids" (denotes the pool we sample concept from), with size of 300 each. Then we perform top-k sampling within each grid to obtain concepts (this process is termed grid sampling). We find this balances the need for sampling top concepts and sample diversity, the variance introduced by small grid facilitates the sampling of more long-tailed concepts in CC-3M.
Crowd-sourcing for challenging visual concepts. During our early inspection, we observed that visual concepts out of CC-3M are mostly about object categories and their relatively simple attributes & relations. However, humans can understand more abstract concepts that require commonsense factual knowledge. Therefore, we further augment the visual concepts with these challenging concepts through crowd-soureing. Specifically, the annotators are instructed write visual concepts by following a predefined set of categories illustrated in Table 1. They are also asked to combine these challenging concepts with those mined from CC-3M. Our experiments confirm that Bongard- Open World problems with these commonsense concepts become more difficult to solve.
2.2 FROM VISUAL CONCEPTS TO REAL-WORLLD BONGARD PROBLEMS
Distractors in positives and hard negatives. To further increase the intra-diversity among the positives P and therefore perplex the induction of given visual concepts C, we prompt ChatGPT to expand it into 10 sentences for positives by inserting distracting objects, attributes, etc. while ensuring common ground is still C. Moreover, prior work on Bongard Problems (Jiang al., 2022)
4

---

Published as a conference paper at ICLR 2024
N
14
Convheeht
OpesCUiP
buse
captioner
PGU heat optiossions
HeaQyeNat
Nradive inago depidtine
Postiivel
Fesesbee clasittotiinn hoad
Negative loaggs deplesting
(opeional) concept is running"
A la dapiciigg
pannttic exgutton?
Positive
(a) Few-shot learning for Bongard- Seeeaard-OpeWWWrrr World.
(b) VLM+LLM (single-ruun)) for Bongard-Open World
cap- p1
cap: p2
malli-round
captioner
cap. 111
cap. m2
wwestion-onswering
VIM
cap. q1
cap. q2
queestooner
ca captioner
LLM
Q1: Deseribe these images detail.
A1: wwo zebras are ranaing the
concept extractor
multi-round
logical operations
LLM
Qu: How many nebras are drinkinng?
Az: Here three zebras
drinking river.
postiiues
cap.
cap.
12 How many beests running?
As: Four wildebeests running
spidate regonties
cap.
cap.
Lot.
Q4: Is there guaas near by river?
A4j: No, the ground full of sand.
cap. q1
cap. 42
Positive
S
Positive, the concept is "animals running..
TeaSuDer
cap.
Negative
(c) VLM+LLM (multi-round) for Bongard-Open (d) Neuro-symbolic approach for Bongard-Open
Figure 3: Models for Bongard-Open World. explore four families of approaches: (a) casting Bongard- Open World into a standard "2-way, 6-shot" fow-shot learning problem and tackling it using few-shot learners with pretrained image representations; (b) combining an LLM (reasoner) and a VLM (image captioner) in single round fashion, where the VLM simply caption each Bongard image and send their captions LLM for solving this problem; extending the method in (b) to multiple rounds, where the LLM will also iteratively probe the VLM for more image details, resulting more condense information solving Bongard; (d) neuro-symbolic approach, where a VLM generates the initial captions, then an LLM extracts visual concepts from them. These concepts are subsequently updated through logical operations, leveraging the responses provided by VLM. until the problem solved. Zoom in for better view.
has shown that improper choices of the negative set A,, e.g. a set of randomly chosen images, could trivialize the challenge by allowing the model to merely recognize part of the visual concepts C (e.g. only recognizing "animals" of full concept "animals are running" in Figure 1) and solve the problem. Therefore, the content of the negative images should overlap with C to ensure the need of inducing full visual concepts. To this end, we again prompt ChatGPT to edit C into 10 sentences for negatives that only partially overlap with it. Both sets will then be used collect images.
Image collection and adversarial query selection. We ask our annotators to pick some sentences from the sets, use an online image search tool find relevant images, and construct the positive World problem by following the principle (the visual concepts C are exclusively depicted by the positives PP. The annotators are then asked to provide two sets of candidates for positive and negative queries based on the unused positive and negative sentences. Finally, we compute the embedding of all images using CLIP (Radford et al., 2021) and the image with maximal embedding distance the mean of positives will be selected as positive query (similar to negative query). We find this adversarial query selection help ensure the difficulty of the problem.
and negative set of a Bongard-Oper Oorrr--ld
Final curation. We employ round of manual refinement as the last hurdle to help curate the dataset. We ask the participant to flag any problem that does not comply with the Bongard principle and provide fix, by replacing the faulty images or modifying the visual concepts C if possible.
5

---

Published as a conference paper at ICLR 2024
2.3 DATA STATISTICS, AND METRICS
Statistics. We present the statistics of Sengard-ppenWordd in Figure 2. Here are some observations: First of all, although our dataset does not assume the structure of the free-form visual concepts C, concepts with lengths 2, 3, and 4 accounts for staggering 90% of all the problems, which both reflect the nature of common visual concepts and avoid overly complicated problems when the concepts become longer. The even distribution of concepts these lengths ensures reasonable difficulty and is verified by our human study in Section 4.2; Morcover, we manage to crowd-source a significant amount of interesting commonsense visual concepts, which make up nearly 1/4 of all the tasks, and each proposed commonsense category has a fair share in the problems; Finally, the word cloud and histogram of concepts demonstrate that some words, egg. "people", "field", etc., can be more popular but the word distribution is long-tailed. This aligns with CC-3M, i.e. lots of human-centered scenes and the concepts indeed follow a long- distribution.
Dataset splits and metrics. We divide the 1.01k Bongard-Open World problems into a 610/200/200 split for reiniig/validationnnss lengths, commonsense vs. non-commonsensse concepts are identical among these three sets. Following the prior work in Bongard Problems (Jiang et al., 2022; Nie et al., 2020), we report the overall accuracy of all models. To help understand the difficulty imposed by the free-form open vocabulary concepts, we further report the accuracy on some subsets: 1) problems with short concepts, í.e. concept length less or equal to three; 2) problems with long concepts. i.e. concept length more than three; 3) problems with commonsense concepts; 4) problems with non-commonsens concepts. These additional metrics provide more detailed and nuanced understanding of the models' performance and can help identify specific strengths and weaknesses of each model. We also introduce a set of metrics that measure the correctness of the visual concepts explicitly induced by the model. Please refer to the Appendix C for detailed deseriptions of these metrics.
We use rejection sampling ensure the distribution of concept
3 MODELS FOR BONGARD-OPENWORE
In general, sets of images P and M, the model ultimately needs to make binary prediction on the query image Ig. A representation network is also needed to process the images and provide input to the few-shot learner. We start by considering several canonical few-shot learners, including non-episodic approaches and meta-learning-bare based avenues. Inspired by the recent progress in Large Language Models (LLMs) and Vision- Language Models (VLMs), we investigate to which extent they can approach Bongard- -oreaddd,, by directly probing VL.Ms, and combining VLMs and LLMs in an interactive reasoning scheme (Zhu et al., 2023; Gao et al., 2022). We also designed a neuro-symbolic reasoning approach that reconciles VLMs and LLMs with logical reasoning. We provide an overview of these methods in Figure 3.
can be viewed as a few-shot learning problem. Given two small
3.1 FEW-SHOT LEARNING FOR HONGARD-OOPNNORRE
Few-shot learning methods. Following the seminal few-shot learning work (Nic et al., 2020; Jiang et al., 2022; Triantafillou al., 2020; Requeima et al., 2019; Bateni et al., 2020), consider meta-learning method, where the learner adopts the episodic learning setting and learns to train a classifier using the support set with meta objective and evaluate the trained classifier on the query. In our experiment, we include the following meta learners: 1) ProtoNet (Snell et al., 2017) and Meta- Baseline (Chen et al., 2020), which both feature a metric-based meta objective; 2) MetaOptNer (Lee et al., 2019) and SNAIL (Mishra et al., 2018), two optimization and memory- based methods. Note that SNAIL uses transformers (Vaswani et al., 2017) and delivers strong results in many few-shot learning tasks and outperforms other baselines on Bongard- Opee-orddd........ World. Readers are encouraged to refer to the Appendix F and the relevant papers for more details.
Auxiliary task of captioning. As we mentioned before, understanding visual concepts serves as the foundation of completing Bongard-Open Worlll tasks. Therefore we investigate if we can boost this via auxiliary task of predicting the concepts. Since we employ free-form open vocabulary concepts, we connect our image encoder to a pretrained BLIP-2 (Li et al., 2023) caption decoder (specifically, the QFormer and the language model) and use caption loss for the task. The overall loss then becomes L = Looggard + CeCapttion where Bongaand and Leuption denote the main loss and the auxiliary loss, respectively. is trading-off weight. More details can be found in the Appendix D.
6

---

Published as a conference paper at ICLR 2024
3.2 COMBINING VLMS AND LLMS FOR BONGARD- OPENWORLD (SINGLE- ROUND)
Evaluation. In this approach, an LLM is used as a few-shot reasoner while a VLM is invoked to translate the images in Bongard Problem into captions. Specifically, we evaluate two publicly available LLMs: ChatGPT (Ouyang et al., 2022) and GPT-4 (Bubeck et al., 2023), and follow the prior practices (Ma et al., 2022b; Gao et al., 2022) to use BLIP-2 (Li et al., 2023) and its successor, InstructBLIP (Dai et al., 2023) to extract captions. Finally, the LLM is anticipated to produce a binary prediction of the query image by learning from the few-shot examples (images as captions) in the prompt through in in-context learning (Brown et al., 2020). We provide more details on the prompt design in the Appendix D.
Explicit visual concepts induction. Since we are particularly interested in the visual concepts induced by the models, further prompt the LLMs to produce explanation of the binary prediction it makes. Specifically, the model is guided to summarize the visual concepts C' that is exclusively depicted by the positíves. The model's summary will then be compared against the ground truth and evaluated using the image captioning metrics we introduced in Section 2.3.
3.3 COMBINING VLMS AND LLMS FOR BONGARD- OPEN WORLD (MULTI-ROUND).
In the previous section, LLM indeed relies on the captions produced by the VLM. However, these captions may lack crucial image information, potentially misleading the LLM reasoner. To mitigate this issue, an additional task can be assigned to the LLM beforehand. This task involves generating questions based on the initial captions and the objectives of BPs. These questions are not hallucina- tions from the LLMs; they are derived from existing information. The VLM utilizes these questions as prompts, providing answers based on the image content, and feedback to the LLM to update existing information. By iteratively repeating this question- guestion-answeringg process multi-round, the LLM can acquire highly detailed information that enhances their reasoning. In our results, we denote this approach as ChatCaptioner, as it is introduced in Zhu et al. (2023).
3.4 A NEURO-S MBBOLIC APPROACH FOR BONGARD- ONNNN-PEENNNN
Bongard- Open World is indeed built upon logical operations visual concepts, which also serve as the foundation for BPs. Inspired by how our humans approach Bongard Problems, we design a neuro- symbolic approach that combines logical reasoning with VLMMs & LLMss. Specifically, after obtaining the captions from a VLM, we leverage GPT-4 to generate meaningful concepts, which currently stands as the most powerful method for semantic understanding. Subsequently, we perform logical operations on the collection of concepts. Then, we update each negative concept with the intersection of positive images and each positive concept with the union of negatives. The intersection of positives must appear in at least four (positives total is six) pictures, although it may be absent in some pictures while still being part of the ground truth concepts. The union of negatives must appear in at least two (negatives total is six) pictures and may partially overlap with positives. We iterate this process until no further updates are required for each image, resulting in the most comprehensive information. Finally, we evaluate the presence of each intersection concept in the query image. If all the concepts exist, indicating that the intersection belongs to the query, it is considered positive; otherwise, is deemed negative. Please refer to Algorithm 2 in the Appendix C for detailed implementation.
EXPERIMENTS
4.1 SETUP
We benchmark the approaches described in Section 3 to evaluate their performances on Bongard- OpenWorld. World. As we mentioned before, all the few-shot learner (excluding ChatGPT and GPT-4)
adopts a ConvNext-base (Liu et al., 2022) image encoder. We consider four pretraining strategies: 1) no pretraining at all (scratch); 2) pretraining with ImageNet-IK IK dataset (IN-IK) (Deng et al., 2009);
3) pretraining with the full ImageNet dataset with more object categories (IN-22K); 4) pretraining with LAION-2B (Cherti et al., 2022; Schuhmann et al., 2022), a massive image-text collection (OpenCLIP). During training, the image encoder will be fine-tuned along with the few-shot learner regardless of being pretrained or not, but use smaller learning rate for the pretrained image encoders. For the auxiliary task, we connect the image encoder to the caption decoder of a pretrained BLIP-2-Opt- 7B model. Note both the QForer and the query tokens are pretrained and trading-off weight of a = 1.0 is used. For LLM- methods, we use the same BLIP-2 and InstructBI.Ir IP model to produce the captions for each image. Thanks the large context length
7

---

Published as a conference paper at ICLR 2024
Table Quantitative results on
the non-LLM models use ConvNeXI-base (Liu et al,, 2022) image encoder, and we experiment with different pretraining strategies: no pretraining at all (scratch), pretraining with ImageNet- IK labels (NN-1K) (Deng et al., 2009), pretraining with full ImageNet-22K labels (IN-22k) and pretraining with LAION-22 2B (Schuhmann al., 2022: Cherti al., 2022) dataset (OpenCLIP). 7 and w/o T indicate
The framework corresponds to the four families approaches depicted in Figure whether the training set of Bongard- Oeee-epeeWWWan World utilized or not. While the LLM-based models use either BLIP-x or ChatCaptioner captions as the image representations. For the auxiliary captioning task, the few-shot learners connected the caption deceder of pretrained BLIP-2-Opt-- B model, and zero-shot models output captions by their reasoning. ddenotes gommongense. ** "involves utilizing the ground truth concepts from Saaedd-ppnWoordt World training and captions from BLIP-2 as inputs fine-tuning ChatGPT over epochs. The fine-tuned model evaluated on the test set. It is worth noting that InstructBLIP was not fine-tuned due to significant drop in its performance on ChatGPT. We splice raw images together in manner similar to the examples in Appendix H. using only query image each time, which then inputted for GPT-4V reasoning.
Bongard-
splits
method
framework
representation
image
aux. task?
short concept
long concept
CS* concept
anything else"
avg.
(a)
IN-22K
59.6
52.7
55.5
56.9
Meta-Baseline
(a)
OpenCLIP
57.8
53.6
55.9
(a)
OpenCLIP
54.6
58.2
55.5
56.6
56.3
(a) W/ T scratch
X
52.3
51.6
54.5
51.0
52.0
(a)
IN-IK
X
60.6
47.3
54.5
54.5
54.5
MetaOptNet
(8)
IN-22K
61.5
53.6
57.9
(a) OpenCLIP
63.3
50.9
60.7
(a) OpenCLIP
62.8
51.8
59.7
(a) w/ T
scratch
X
57.8
50.5
48.2
56.9
54.5
(8)
IN-IK
56.9
54.9
51.8
57.6
ProtoNet
(8)
IN-22K
X
62.4
51.6
54.5
58.6
(a) OpenCLIP
61.9
59.1
57.9
(a) OpenCLIP
59.2
$7.7
51.8
61.0
58.5
(a) w/ T scratch
X
52.8
46.2
50.9
49.3
49.8
(a)
IN-IK
X
61.5
54.9
48.2
62.4
58.5
SNAIL
(a)
IN-22K
62.8
57.7
54.5
62.8
(a) OpenCLIP
64.2
$7.3
62.8
(a)
OpenCLIP
66.1
61.5
63.6
64.1
64.0
Openinamingo
(b) w/o
OpenCIIP
50.0
48.4
50.9
48.6
49.3
Otter
(b) w/o T
OpenCLIP
49.3
49,3
48.9
49.4
49.3
w/o T
BI.IP-2
60.6
56.6
55.5
60.0
58.8
ChatGPT
(b) w/o T
InstructBLIP
52.1
50.6
48.1
52.7
51.4
(c) w/o T
ChatCaptioner
52.3
45.6
57.3
46.2
49.3
ChatGPT (Fine-tuned))"
(b) w/ T
BLIP-2
67.0
58.8
55.5
66.2
63.3
w/o
BLIP-2
64.5
57.3
63.2
GPT-4
InstrucBLII
67.3
59.3
65.6
GPT-4V
(b) w/o T
Raw Images
/
54.6
53.3
50.9
55.2
54.0
Neuro- Syno--yymoolic
(d) w/o T
InstructLIIP
58.3
52.2
56.4
55.2
55.5
Human
N/A
N/A
N/A
91.7
90.1
89.1
91.7
91.0
allowed by the GPT-x family, no downsampling of captions is needed. Finally, we select the best model on the validation and report its metrics on the test proposed in Section 2.3. Full details on the hyperparameters can be found in the Appendix D.
4.2 ANALYSIS AND INSIGHTS
We provide the full quantitative results of the considered models (detailed in Section 3) on Bongard- Open World in Table 3. The major findings are summarized below:
Challenges of free-form visual concepts. Table 3, we demonstrate both overall averaged accuracy on the test set and also four types of splits introduced in Section 2.3. It can be observed that once a model does better than coin-flipping (50% chance of success), it generally struggles more on Bongard- World problems with longer free-form concepts versus shorter ones. Also, problems with abstract commonsense visual concepts (as described in Section 2.1) impose a greater challenge than those without it. This aligns with hypothesis that few-shot reasoning with free-form visual concepts is indeed more difficult, compared to counterparts with a fixed set of image categories or HOIs, which relatively short and likely do not include any commonsense aspects.
8

---

Published as a conference paper at ICLR 2024
Representations and open vocabulary. As mentioned before, reasoning with Bongard- OpenWorld problems requires the models to embrace an open vocabulary setting, where the set of all possible visual concepts are not provided priori. Our experiments with different image representations verify that, by making the pretraining strategy closer to open vocabulary, i.e. ex- panding the vocabulary of visual concepts the image encoder exposed to (in our case, from none to IN-IK, IN-22k, and LAION-2B label set), most of the evaluated few-shot learners can attain consistent improvement. Note that even with the fully open vocabulary pretraining (OpenCLIP), the gap between the best model (SNAIL) and humans is still quite substantial, implying that better pretraining is not a complete solution but more of foundation for solving the reasoning task in Nongard-OpenWorlee
The role of captioning. Table 2, we have shown that the free-form visual concepts Bongard- Open World are image captions. Therefore, we are interested in exploring to which extent the captioning task itself can help with Longard-OpenWortee
First of all, we can observe that adding the auxiliary captioning task can slightly boost ProtoNet (+0.2%) while escalating Meta-Baseling (+2.5%) and SNAIL (+2.7%) more significantly. We also find fine-tuning with the auxiliary task could be more challenging as large model (BLIP-2 decoder) is plumbed to the learner, making some few-shot learners unable to benefit from the auxiliary task.
On the other hand, even with off- fr-thhe-sheff captions from BLIP-2, 22, LLM-based methods still fail to close the numan-machine gap. We hypothesize the reason to be two-fold: 1) distractors, without awareness of the Bongard- Seerardd-peWWorrrr World task and other images in the current problem, the captions BLIP-2 produces could be distracted by irrelevant content and miss description needed by the visual concepts: 2) hard negatives, as illustrated in Figure 1, the conceptual similarity between positives and negatives could perplex the concept induction of the LLM. Due to space limitation, we defer qualitative results of explicit concept induction of LLM- -bseed method to the Appendix G.
Different few-shot learners. Many previous benchmarks (Nie et al., 2020; Jiang et al., 2022) have demonstrated that some few-shot learners are generally better than others in few-shot visual reasoning. Our winner here is SNAIL,, a memory-based learner with transformer architecture. It even surpasses the strong GPT-4 baseline on all measures. We hypothesize that the memory -based approach could suit Bongard-Open World better our dataset does not offer huge amount of few-shot training episodes. Therefore, additional inductive biases like the sample retrieval in SNAIL could make the learner more data efficient.
Limitations of current VLMs. The results in Table 3 indicate that combining VLMs & LLMs in zero-shot fashion, specifically using InstructBLIP to generate more complex captions, leads to significant drop in the performance of ChatGPT. We hypothesize this drop is due to InstructBLII introducing excessive noise that interferes with the reasoning of ChatGPT, while the robustness of GPT-4 remains unaffected. We attribute this to the limitations of current caption model in accurately representing open vocabulary free-form visual concepts, abstract visual attributes, and commonsense factual knowledge in Bongard-Open World. Additionally, even powerful end-to-end pre-trained VLMs (e.g., OpenFlamingo, Otter) still face challenges with multi mullti-maage reasoning, which is a unique and particularly difficult aspect of Bongard Bongard-OpenWeir
Neuro-Symbolic also failed. While performing logical operations directly on visual concepts may initially appear an ideal solution, but the results are not satisfactory. GPT-4's ability to comprehend and extract concepts unquestionable. However, we observed that the accuracy of inducing the true concept is disappointingly low, with significant presence of irrelevant noise concepts, due to the inability of the concept extractor, i.e. the VLM. This leads us to point that, as mentioned earlier, developing powerful model capable of multi-image reasoning and accurately inducing open vocabulary free-form visual concepts still requires tremendous effort.
CONCLUSION
We have introduced for machine vision. It combines the best of few-shot learning and complicated real-world visual concepts. The diverse nature of these concepts including abstract visual attributes and commonsense factual knowledge impose great challenges to AI. oftte--o-the-rtt fhw-shot learners and even sophisti- cated systems reconcile LLMs & VLMs, largely fall behind human contestants. We invite people from different AI communities to join our challenge for this grand challenge.
a benchmark for evaluating real-worle few-shot reasoning

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We extend our heartfelt gratitude to the anonymous reviewers whose dedication and insightful feedback have significantly enhanced the caliber of this paper. Their constructive critiques and valuable suggestions were instrumental in refining our work. Additionally, we are deeply appreciative of the Program Chairs and Area Chairs for their meticulous handling of our submission and for their comprehensive and invaluable feedback. Their guidance has been pivotal in elevating the quality of our research.
This work is supported by National Key R&D Program of China (2022ZD0114900) and in part by National Natural Science Foundation China (Grant No.61976214).
REFERENCES
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: new benchmark for physical reasoning. In Advances Neural Information Processing Systems, pp. 5003-509344 2019. 22
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005. 15
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning, pp. 511-520. PMLR, 2018. 2.
Peyman Bateni, Raghav Goyal, Vaden Masrani. Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14493-14502 2020. 6
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 33:1877-1901, 2020. 7, 20
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv: 2023.
Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. HICO: A benchmark for recognizing human-object interactions in images. In ICCV, 2015. 2
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. new meta-baseline for few-shot learning. arXiv preprint arXiv:2003. 043900 2020. 6
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive togguage-image learning. arXiv preprint arXiv:2212.07144 2022. 7,8
François Chollet. On the measure of intelligence. arXiv preprint
2019. 22
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. 7
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 258-255. IEEE, 2009. 7, 8, 20
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018. 20
10

---

Published as a conference paper at ICLR 2024
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (IJCV), 88 (2):303-338) 2010. 20
Li Fe-Fei et al. A bayesian approach to unsupervised one-shot learning of object categories. In Proceedings Ninth IEEE International Conference on Computer Vision, pp. 1134-1141. IEEE, 2003. 20
Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. Natural Language-Centrls Outside- Knoiid-enowledge Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5067-5077, 2022. 3, 6,7
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of International Conference Computer Vision (ICCV), 2017. 1
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001. 20
Siyuan Huang. Humar-like Holistic 3D Scene Understanding. University of California, Los Angeles, 2021.
Drew A Hudson and Christopher D Manning. Gqa: A dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Xiaojian Ma Huaizu Jiang, Weili Nie, Zhiding Yu, Yuke Zhu, Song-Chun Zhu, and Anima Anand- kumar. Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Conference Computer Vision and Pattern Recognition (CVPR), 2022. 1, 4, 6, 9, 22
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017, 22
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et Segment anything. arXiv preprint 3.XY::2304.02643., 2023. 20
Ranjay Krishna, Yuke Zhu andd Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1):32-73, 2017.
Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-ylm: Open-vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209. 2022. 20
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science. 350(6266): 132--1633)333-333883 2015. 1, 1,4, 21
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019. 6. 20
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201 033566, 2022, 20

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping ergguag--imgg pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301 12597. 2023. 1, 6, 20
Yonglu Li, Liang Xu, Xijie Huang, Xinpeng Liu, Ze Ma, Mingyang Chen, Shiyi Wang, Haoshu Fang. and Cewu Lu. HAKE: human activity knowledge engine. CoRR, abs/ sess1994406639, 2019. 2
Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004. 15
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014. 1, 20
Zhuang Liu, Hanzi Mao, Thao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976 1986,, 2022. 7,8
Xiaojian Ma, Weili Nie, Zhiding Yu, Huaizu Jiang, Chaowei Xiao, Yuke Zhu, Song-Chun Zhu, and Anima Anandkumar. Relvit: cancept-guided vision transformer for visual relational reasoning. arXiv preprint 2rXiv:2204.11 2022a. 1, 22
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022b.
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010. 1, 22
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. ICLR, 2018. 3, 6, 20
Weili Nie. Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. Bongard-logee new benchmark for human- concept learning and reasoning. Advances in Neural Information Processing Systems, 33, 2020. 1, 2, 4, 6, 22
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2002. 15
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021. 3, 5,20
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. ICLR, 2020. 21
Mengye Ren, Eleni Triantafillou, Kuan-Chieh Wang, James Lucas, Jake Snell, Xaq Pitkow, Andreas Tolias, and Richard Zemel. Probing few-shot generalization with attributes. arXiv preprint arXiv:2012.05889, 2020. 2
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, 32, 2019. 6
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with -ugmentd neural networks. In International conference on machine learning, pp. 1842-188 1850, 2016. 20
12

---

Published as a conference paper at ICLR 2024
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. arXiv preprint arXiv:1904.01533 2019. 22
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint Stt::2220000800,, 2022. 7.8
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning, In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2556-25655 2018. 2, 4
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical Networks for Few-shot Learning. Advances in Neural Information Processing Systems, 2017. 6, 20
Damien Teney, Peng Wang. Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel. V-prom: A benchmark for visual reasoning using visual progressive matrices. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1207 1-1277, 2020. 4, 22
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In ICLR, 2020. 1, 4
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017. 6
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-45777 2015. 15
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NeurlPS, pp. 3630-3638, 2016. 1, 4, 20, 21
Erik Weitnauer and Helge Ritter. Physical bongard problems. Ifip international conference on artificial intelligence applications and innovations, pp. 157-163. Springer, 2012. 22
Sirui Xie, Xiaojian Ma, Peiyu Yu, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Halma: Humanlike abstraction learning meets affordance in rapid problem solving. arXiv preprint arXiv:2102.11344 2021. 22
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048-2052 PMLR, 2015.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint 1rXiv:1910.01442. 2019. 22
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14393 14402, 2021. 20
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chus Zhu. Raven: dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5317-5327, 2019. 1, 22
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elho- seiny. Chatgpt asks, b answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.06594 2023. 3, 6, 7,
Song-Ch Chum Zhu and Yixin Zhu. Cognitive Models for Visual Commonsense Reasoning. Springer, 2021. 1, 20, 22
13

---

Published as a conference paper at ICLR 2024
A LIMITATION STATEMENT
We carefully curate Bongard-Open World to encompass open vocabulary and free-form visual con- cepts, which present exceptional challenges for current VLMs. We have investigated range of approaches, including methods based on CLIP pre-training and ConyNeXttbasee image encoder, combining pre-trained caption models with the GPT-x family, as well as incorporating interactive answering systems in single-round or multi-round fashion, even the ceuro-symbolic ap- proach performs logical operations on visual concepts. However, all of these attempts have yielded unsatisfactory results, and we have not discovered a promising solution. Nevertheless, we remain dedicated to further exploration and experimentation. For now, we propose that models capable of simultaneous multi-imaae reasoning hold the potential to tackle Bongard-Open World challenge.
B BROADER DISCUSSION
We would like to emphasize that, adding open-world concepts is the basic requirement of few-shot visual reasoning benchmark can be still relevant in the VLM era today. However, being able to handle open-world concepts like Tlamingo/ChaiGP//OOee does not mean the model can solve Bongard- OpenWorld. Because there one thing that could still be missing in these models: reasoning-aware perception and holistic perception-reusoning
Separately done perception and reasoning cannot beat Bongard-Open Oordldldd, no matter how powerful the models are.
Our benchmark requires extracting common visual concepts globally, rather than extracting atomic concepts from each image separately. The idea is, images in Bongard-Open World have rich details, and therefore it is almost impossible to convert everything the images depicted into text and let LLM or other logic reasoner analyze what is being shared by positives and not depicted by negatives. Instead, VLM-based captioners usually fail to capture the image contents relevant to the current Bongard Problems (see Figure in Appendix G, VI.M is distracted by the cardinal and therefore did not descrihe the concept that is useful for reasoning. which is "tree or branch covered in snow"), therefore leading the follow-up reasoning with LLM fail.
Holistic perception-reasonig is hard, and it is true for largely pretrained VLMs (as for autumn 2023).
An ideal model for Bongard-Opent OpenWorld World will need to compare images from both positives and negatives, then figure out what is the right concept for classifying query image, i.e. reasoning-aware perception holistic perception and reasoning. We, therefore, try our best to find open-sourced VLM that has claimed such capability. Please note that the basic requirement will be handling multiple images the same time. which already rules out most of the publically available VLMs. We ends up trying OpenFlamingo, Otter and GPT-4V. Clearly, even the latest and most powerful GPT-4V failed to close the numan-machinn gap as well.
The above reasons indeed highlight what might be going wrong with ChatGPT, which relies on seperate VLM captioner, and Plamingo/OOter//P11A4 which could fall short on perform the reasoning required by Bongard-Open World. On the contrary, all the few-shot learning baselines (from ProtoNet SNAIL) have been trained on the training set of Bongard-Open World, therefore they have learned (not perfect though) to perform the reasoning needed here, while also equipped with worn-world or vocabuuaay perception thanks to the OpenCLIP backbone. Therefore, we can see the gap between the best FSL method and these zero-shot large models.
C MORE DETAILS ON BUILDING BONGARD-OPENWOnk
Grid-sampling of visual concepts from CC-3M. We streamline all the captions from CC-3M into concept tuples and perform grid sampling. Its key operations to extract visual concepts from the tagging
streamlined captions and pseudo code shown in Algorithm 1: 1) We run part-of- (POST) and sliding window with size 2/3/4/5 over all the concept tuples to count the frequency of concept with length 2/3/4/5; 2) Instead of picking the top concept tuples of the whole CC-3M dataset,
14

---

Published as a conference paper at ICLR 2024
Algorithm 1 Grid sampling visual concepts C' from CC3M
Input: CC-3M caption grids Grids, candidate lexicals Leas, top factor k. Output: Candidates for concept tuples Q.
1: Q
2: for = 1 to len (Grids) do
3: grid Grids[i];
4: q= 0:
5:
for j= len(grid) do
6:
caption = gridjj];
7:
tuple POST(cappiion)
8:
tuple filter(tuple, key = Lexs);
9:
tuple s window(ttplee);
10:
q q tuple;
11:
end for
12:
g top (g, K = k);
13:
Q= QUg:
14:
end for
15: return Q:
each time we only compute the frequency within a grid with only 300 (grid size) CC-3M concepts, pick the top k (k=3) candidates for concepts C, then move to the next grid. We find this balances the need for sampling top concepts and sample diversity, as the variance introduced by a small grid facilitates the sampling of more long-tailed concepts in large dataset like CC-3M.
Crowd-sourcing for challenging visual concepts. During our early inspection, we observed that visual concepts of CC-3M are mostly about object categories and their relatively simple attributes & relations. However, humans can understand more abstract concepts that require commonsense factual knowledge. Therefore, we further augment the visual concepts with these challenging concepts through crowd-sourcing. Specifically, the annotators are instructed to write visual concepts by following a predefined set of categories illustrated in Table 9, more examples of non-commonsense and commonsense visual concepts are shown, including the ratio of each concept category in Bongard- Wornd They are also asked to combine these challenging concepts with those mined from Worn CC-3M. Our experiments confirm that Bongard-Open World problems with these commonsense concepts become more difficult to solve.
Caption metrics. We also introduce set of metries that measure the correctness of the visual concepts explicitly induced by the model. As we mentioned before, the model will produce a sentence, i.e. image caption, which will then be compared against the ground truth. We use standard metrics for captioning tasks, including BLEU1/2/3/a (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005). ROUGE, (Lin, 2004) and CIDEr (Vedantam al., 2015).
Explicit caption induction. These caption metrics provide objective measurements for assessing the quality and effectiveness of machine translation, text summarization, and image captioning systems. In our experiments, we evaluate two publicly available LLMs: ChatGPT and GPT-4, Since the public version of both models does not support multi-modal input, e.g. images, we implement the representation function repr(-) above as image captioning. Specifically, we ask BLIP-2 and InstructBLIP to produce the most-probable caption for all images in PUN and the query I.. We also consider a variant that uses the URL of an image, which could leak some information about its content, as the representation. Since we are particularly interested in the visual concepts induced by the models, we further prompt the LLM to produce explanation of the binary prediction makes. Specifically, the model is guided to summarize the visual concepts that are exclusively depicted by the positives. The model's 's summary Ć will then be compared against the ground truth and evaluated using the image captioning metrics introduced above.
Logical operations on visual concepts. We have presented comprehensive description of logical operations in Section 3.4, the query concept will be updated by final intersection of positives.
15

---

Published as a conference paper at ICLR 2024
Algorithm 2 Logical operations visual concepts
Input: Positive concepts Cpt Positives Negative concepts Opt Negatines Query concepts Qpiquery
Output: Intersection positive concepts Inttersection
1: while True do
2:
IptIntereection = Ni=1 Cpt
Positice
3:
CptInteraection
SilteerCCppr Intersection value >= 4);
4:
Cplunion U Cpl Negatioe"
5:
Optúnion f filter (ptu value >= 2):
6:
Cpt C Ctunioo Cpt Positivei
CaalaateCCl.osstttiiie Cpt Cpt diff
7:
Cpt
Positive
Opt Negatsue Inteerecctionn C't
8:
9:
Netrtigtive update Opt Negatioes diff
Cpt
fiff
Negative
10:
if Cpt Positivn 0 & Cpt
0 then Negation
11:
break;
end if
13:
end while
df = Cpt Intersection - Queru
14:
Cptquery update(Cptqquryy Cpt Query):
return Cptrntersection
Exact instructions for annotators. When annotators are instructed to write visual concepts by following predefined set of categories illustrated in Table and asked to combine these challenging concepts with those mined from CC-3M, here are the guides to our annotators:
General concept annotation guide:
1. Positive images all share concrete concept, denoted C. This concept should be engaging and not overly simplistic (such as basic objects, 'dog' or "aat', but rather somewhat abstract (for instance, 'running' 'reflection)..
2. Each negative image must depict concept that must have some added distractors with C.
3. Each negative image must depict concept that partially overlaps with C.
Commonsense annotation guide:
Follow the definition of commonsense, manually revise the automatically collected candidate concepts (from CC-3M) to make them more "commonsense" allike.
One definition of commonsense from Wikipedia: "Commonsense knowledge includes the basic facts about events (including actions) and their effects, facts about knowledge and how it is obtained, facts about beliefs and desires. It also includes the basic facts about material objects and their properties." The following are some examples:
1. HOI: discerning subtle interaction between humans and objects, c.g. playing basketball vs. playing volleyball.
2. Taste Nutrition Food: identifying ingredients in food, e.g. high-calorie vs. high-sugar.
3. Color / Material / Shape: differentiating similar materials with various appearances, e.g. rough
wood vs. painted wood.
4. Functionality / Status / Affordance: recognizing variants that have the same function, e.g. cuttable.
5. And Or Not: making logical judgments about something, e.g. something.
6. Factual Knowledge: States.
possessing factual knowledge, e.g. buildings in the capital of the United
induction of metaclass, e.g. rescue vehicle (including ambulance, police car, fire engine, etc), Felidae (including cat, lion, puma, etc).
7. Meta Class:
16

---

Published as a conference paper at ICLR 2024
8. Relationship: perceiving spatial relationships, e.g. "near", which may solely be "near pixel distance" but located in the foreground and background respectively results in "far on physical distance".
9. Unusual observations: e.g. light refraction, gaze direction.
As you can see, the annotators are guided to revise & reorganize the original CC-3M concepts into commonsense concepts, with explicit guidance on what can be viewed as commonsense concepts. This aligns with the descriptions in the main paper.
D IMPLEMENTATION DETAILS AND HYPER-PARAMETERS
Few-shot learning formulation. Here we offer formal definition of few-shot learning task in Bongard-Opennee OpenWorld. Recall the formulation in Section 2, each of our Bongard problem instance (C, CC,,,N,,, In) comprises one few-shot learning episode with N = 2 classes and 2M samples. There- fore the learner has to learn from a set S PUN = {(T,: 1)..... (T 1), (T O),... (T 0)} and is evaluated on query image (Lgr H1g). Each example (1, y) includes image I #x
and class label y E f0, 1}. indicating if the visual concepts C (exclusively depicted by P) presents I. We make M in Bongard- OpeendOOpnnnnne World and therefore it becomes "2-way, 6-shot" few-shot learning episode.
Distractors positives and hard negatives. further increase the intra-diversity among the positives P and therefore perplex the induction of visual concepts, given C, we prompt ChatGPT to expand into 10 sentences for the positives by inserting "distracting" objects, attributes, etc. while ensuring their common ground is still C. Moreover, prior work Bongard Problems has shown that improper choices of the negative set N, e.g. a set of randomly chosen images, could trivialize the challenge by allowing the model to merely recognize part of the visual concepts (e.g- only recognizing "animals" of a full concept "animals are running") and solve the problem. Therefore, the content of the negative images should overlap with C ensure the need of inducing full visual concepts. To this end, we again prompt ChatGPT to edit C into 10 sentences for the negatives that only partially overlap with it. Both of sentences will then be used to collect images. For all models, we invoke the latest version from OpenAl and set "temperature" as 1.0.
We the ChatCompletion API of gpt- 00. 5-tubb model and following prompt:
Given a concept such as "concept: bird fly". Then, Please help me to complete the following
2 tasks:
Expand the "concept" into 10 positive sentences, their semantics are different but both share all words in the "concept". For example, positive sentences could be "positive: [a bird fly in the sky, a bird fly in the rainy day. the large bird fly above clouds, a bird fly over the river, bird fly over mountains, some birds fly in V formation, birds in colorful sky, bird fly icon, bird fly papercut, cartoon illustration of bird fly]".
flying in
2. Reduce the "concept" into 10 negative sentences, their semantics different but both missing only one word in the "concept". For example, negative sentences could be "negative: [a bird eating seeds, bird singing on wire, a bird building nest, ladybug fly in the rainy day, a bird perched on a branch, bird pecking at the ground, an airplane fly over mountains, bird trapped in a cage, bird specimen, fly fish]".
Please complete the following example:
VLM LLM for Songard-OpenWorld. We evaluate two publicly available LLMs: ChatGPT and GPT-4, both in single-round and multi-round fashion. We ask BLIP-2 and InstructBLIP to produce the most- prost-probbble caption for all images in S = PUW and the query Ig. For the multi-round fashion, followed the ChatCaptioner (Zhu et al., 2023) prompt design.
For the single-round fashion, we use the ChatCompletion API of gpt-3. 5-turbo and gpt-4 model and following prompt:
17

---

Published as a conference paper at ICLR 2024
Given 6 "positive" sentences and 6 "negative" sentences, where "positive" sentences be summarized as 1 "common" sentence and "negative" sentences cannot, the "common" sentence describes
of concepts that are common to "positive" sentences, And then given 1 "query" sentence, please determine whether the "query" belongs to "positive" or "negative" and give the "common" sentence from "positive" sentences.
Please complete the following query:
End-to-end pre-trained VLMs for Bongard-OpenWWrrt In our VLMs experiments, we evaluate two publicly available VL.Ms: OpenFlamingo and Otter, we use the same prompt as following:
Given 6 "positive" images and 6 "negative" images, where "positive" images share "common" visual concepts and "negative" images cannot, the "common" visual concepts exclusively depicted by the "positive" images. And then given 1 "query" image, please determine whether belongs to "positive" or "negative".
"positive" oitercaceillltuuuuomeeeeeeeeeeee "negative" /inpa//swicttititiimmmmmmmmmmimn "query" image:dlondochuutb>cmaegs "query" image belongs to
Neuro-symbolic approach for Bongard-Open World. We use GPT-4 to extract meaningful concepts from caption sentences, the prompt as following:
Given sentence that describes set of visual concepts, these concepts may be Adjectives, Nouns, Verbs, or Adverbs. Please identify these concepts and organize them into a Python list.
We employ GPT-4V for reasoning on legends similar those in Appendix H, using single query image for each instance, the prompt as following:
This image layout consists of three rows. The first row displays six positive images that all depict one shared visual concept, and the second row displays six negative images none of which depict this concept. The third row features a single query image.
Please look very carefully for details of all images and perform contextual reasoning, deter- mine whether query belongs positive negative, and summarize the shared visual concept in sentence only includes unique core visual concepts.
WARNING: the query belongs the positive only when it depicts the shared concept; otherwise it belongs to the negative.
WARNING: please look very carefully and reexamine your result before answering. positive or negative: the shared visual concept:
Few-shot learning for Bongard-Open World. Main hyperparameters of each few-shot model in our experiments shown in Table 8.
Wornaldd.
E
ADDITIONAL EMPIRICAL RESULTS
E. EXPLICIT VISUAL CONCEPTS INDUCTION ON HONGARD-OOPNNORR
Due to space limitation, we defer quantitative results explicit concepts induction of LLM-based method to here. We use the aforementioned caption metrics to measure the performance of LLM- based models on induction of visual concepts. As depicted in Table 4, GPT-4 exhibits not only 2.8% higher binary prediction accuracy compared to ChatGPT but also demonstrates notable improvement in explicit visual concepts induction. We speculate that the performance of LLM may be detrimentally affected by the quality of captions generated by VL.M.
18

---

Published as a conference paper at ICLR 2024
Table Explicit visual concepts induction Bongard-Open We prompt the LLM-based approaches explicitly summarize the visual concepts the positives and evaluate their output based on captioning metrics introduced in Appendix C.
method
representation
BLEU;
BLEU2
BLEU;
BLEU,
METEOR
ROUGE,
CIDEr
BLIP-2
0.186
0.109
0,070
0.045
0.154
0.258
0.781
ChatGPT
BLIP-2 w/ Fine-tuning
0.441
0.292
0.209
0.153
0.222
0.417
1.714
InstructBLIP
0.111
0,060
0.034
0.017
0.135
0.182
0.198
BLIP-2 0.310
0.199
0.140
0.100
0.207
0.358
1.351
GPT-
InstructBLIP
0.136
0.075
0.045
0.027
0.154
0.202
0.256
GPT-4V
Raw Images
0.190
0.073
0.029
0.023
0,111
0.188
0.527
E.2
RESULTS COMPARED TO RELEVANT BENCHMARK (BONGARD-HOOL
Compared to Bongard -OOI, there are both shared observations and differences.
1. shared observations: canonical FSL (ex. SNAIL, MetaBaseine) do not work well on both datasets, regardless what representation backbone being used.
2. differences:
the most effective methods differ (Bongard-OpenWorld is SNAIL vs. Bongard-HOI is Meta- Baseline);
Bongard-Hon HOI is more sensitive to open-vocabulary pretrained representations (its concept vocabulary limited); Bongard-Open Srggard-OpenWorldd World not as is designed to be open vocabulary or open-world;
Bongard-HOO -OI does not have results of LLM- methods.
The first and last observations are obvious. Here we would like to elaborate a on the second one. We offered some additional results of benchmarking CLIP (shown below), relatively more free- form and open-world visual backbone on both Bongard-HOO (a prior work, focus on compositional generalization closed-world concepts) and Bongard-Openworld. As you can see, with much stronger backbone like OpenCLIP, ProtoNet can achieve 71.3% acc on Bongard-HOI (way higher than the previous Bongard- -OI SOTA 58.3% using ImageNet-Il 1k pretrained backbone), but Bongard-Oper World it can only reach 58.3%. In fact, Bongard-HO discourages the use of such an open-world pretrained backbone like CLIP because it might undermine the challenges posed by the benchmark (compositional generalization will not be an issue anymore, as all concepts in training/generaiization test have been saw by CLIP). Rather, Bongard- World allows and indeed encourages all types of backbone and models by introducing free-form open opendworld concepts. And as the additional result suggest, we're 'r're successful even the powerful CLIP backbone is still far from beating our benchmark.
Table 5: Results compared to Bongard-HOO. With much stronger backbone like OpenCLIP, Proto achieve 71.3% Bongard-HOI (way higher than the previous Bongard-HOI SOTA 58.3% using ImageNet- Ik pretrained backbone), but Kongard-OOennWarn can only reach 58.3%.
benchmark
method
image representation
aux. task?
avg.
Bongard-Opent World
ProtoNet
ImageNet-Ik OpenCLIP
X 56.0
Bongard-OpenWorld
ProtoNet
X
58.3
Bongard-HOr
ProtoNet
ImageNet-Ik
X
58.3
Bongard-HOI
ProtoNet
OpenCLIP
x
71.3
E.3 IMBALANCE BETWEEN POSITIVE AND NEGATIVE QUERIES.
This section presents further results singling out the accuracy on positive and negative queries. Table shows that there might be slight difference for single model, but did not observe pattern or preference towards positive or negative queries globally. Therefore Bongard-OpenWoin World does not appear to exhibit strong imbalance issue.
19

---

Published as a conference paper at ICLR 2024
Table 6: Results of imbalance between positive and negative queries. Each result is derived from the most effective model corresponding each method.
method
image representation
aux, task?
positive query
negative query
avg.
OpenCLIP
59.5
53.0
56.3
MetaOptNet
OpenCLIP
57.9
57.0
57.5
ProtoNet
OpenCLIP
58.2
58.7
58.5
SNAIL
OpenCLIP
61.2
66.8
64.0
GPT-4
InstructBI..
67.4
60.2
63.8
E.4 RESULTS NATURAL BASELINE.
In fact, natural baseline could be very similar to ProtoNet (which we already benchmarked), but ProtoNet further fine-tunes the visual features on the training set. This how we implement it: we calculates the mean similarity between query image embedding (produced by CLIP or DINO) and both positive and negative images embeddings (also produced by CLIP or DINO) in the support set, assigning the label of the higher mean its prediction.
Table reveals that natural baseline with both CLIP and DINO as backbone fail to produce comparable performances the existing baselines. Our hypothesis aligns with the initial analysis with CLIP, when constructing Bongard- -pen and negative images in order make the problem harder. Although the similarity measured with CLIP embedding disantance, as the results above suggested, embeddings based on other approaches, ex. DINO and DINO-v2 could also be affected.
OpenWorld, World, we delibrately increase the similarity between positive queries
Table 7: Results of natural baseline. The model architecture for CLIP is VIT-H//14 which reached a performance of 78.0% on the ImageNet top-1 accuracy. The model architecture for DINO is ViT-B/8, which achieved a performance of 80. % the ImageNet top-1 accuracy. The model architecture for DINOv2 is ViT-g/14 14 (with registers), achieving performance of 87.1% on the ImageNet top-1 accuracy.
method
image representation
task?
short concept
long concept
CS' concept
anything else
uvg.
OpenCLIP
X
12.8
7.7
12.7
9,7
10.5
natural buseline
DINO
X
18.3
11.0
16.4
14.5
15.0
DINOv2
X
17.9
11.0
14.5
14.8
14.8
ProtoNet
OpenCLIP
/ 59.2
57.7
51.8
61.0
58.5
F RELATED WORK
Free-form and open vocabulary computer vision. Historically, due to the limitation on data annotation and model training, many of the computer vision benchmarks only adopt a small set of visual concepts, e.g. PASCAL VOC (Everingham et al., 2010), Microsoft COCO (Lin et al., 2014) and even the most popular split of ImageNet (Deng et al., 2009) only includes IK object categories. The recent surge of pretraining and open-ended models like LLMs (Devlin et al., 2018; Brown et al., 2020) has called for the free-form open- voccn-ocabulary vision models and benchmarks (Kuo et al., 2022; Kirillov al., 2023; Zareian al., 2021; Li al., 2022; Radford et al., 2021). While the goal is to accommodate any visual concepts, most of these works so far only serve the task mid or high-level vision, including language-driven image recognition and object detection/seemmenttioon Our Bongard- Open World benchmark fills in the blank of open vocabulary in visual reasoning and few-shot learning. To solve our tasks, these models have to go beyond recognition, step into reasoning, and induce open- open-world visual concepts. Our results with ofate-of-the-ae art open vocabulary recognition model like BLIP-2 2 (Li et al., 2023) and CLIP (Radford al., 2021) also confirms that Bongard Ooeddd-pppnllll cannot be trivially solved by these models.
Few-shot and meta learning methods. Few-shot learning studies learning from limited number of examples (Fe-Fei et al., 2003; Zhu & Zhu, 2021). In the past decade, meta-learning (or learning- to-learn) (Hochreiter et al., 2001) has become a leading family of approaches to tackle the few-shot learning challenge. The main idea is to discover the generic knowledge across tasks to adapt to new task quickly. Several meta-learning methods have been developed and can be categorized into three types: 1) memory-based methods, such as MANN (Santoro et al., 2016) and SNAIL (Mishra et al., 2018); 2) metric- based methods, such as Matching Networks (Vinyals et al., 2016) and Pro- toNet (Snell et al., 2017), and 3) optimization-base methods, such as MetaOptNet (Lee al., 2019)
20

---

Published as a conference paper at ICLR 2024
Table 8: Hyperparameters for the few-shot learning models. The MetaOptNet models shows experimental results for scratch, IN-IK. IN-22K, OpenCLIP and SemnCLIPCCapiinnn The hyperparameters scratch, IN-IK and IN-22K options for other few-shot models are consistent with MetaOptNet, therefore all of them are omitted, showcased are their own unique hyperparameters.
Parameter
Value
MetaOptNet
Optimizer
AdamW le
Learning rate
Weight decay factor
5e
Learning rate schedule
Warmup cosine
Total training epochs
20
Batch size
10
Image encoder learning rate
1e-5
MetaOpiNet (Caption Only)
Caption head
OPT captioner
Caption LM
facebook/opt-125m
Use qformer
True
Pretrained qformer
True
Freeze qformer
False
Pretrained query
True
Freeze query
False
Use unpooled
True
Caption loss coefficient
0.1
Meta-Baseline
Image encoder learning rate
le-4
Mela-Baseline (Caption Only*)
Freeze qformer
True
Freeze query
True
Caption loss coefficient
0.1
ProtoNet
Image encoder learning rate
3e
ProtoNet (Caption Only)
Freeze qformer
True
Freeze query
True
Caption loss coefficient
1.0
SNAIL
Image encoder learning rate
le-4
SNAIL (Caption Only)
Freeze qformer
False
Freeze query
False
Caption loss coefficient
1.0
and ANIL (Raghu et al., 2020). These approaches have achieved excellent performances on existing few-shot learning benchmarks such as minilmageNet (Vinyals et al., 2016) and Omniglot (Lake et al., 2015). However, these benchmarks are primarily focusing image recognition with fixed and relatively small set of visual concepts (mostly object categories), i.e. they fail to account for the challenging free-form and open vocabulary visual concepts in few-shot learning setting. We believe our Bongard-Open World can help fill this gap and foster the emergence of better few-shot learning methods for open-wond world free-form visual concepts.
21

---

Published as a conference paper at ICLR 2024
Visual reasoning benchmarks. Abstract visual reasoning is believed to be capability of human-level visual intelligence (Marr, 2010; Zhu & Zhu, 2021; ; Ma et al., 2022a) and therefore several benchmarks have been developed to measure the progress of current Al it. Notably examples include compositional question answering (Johnson et al., 2017; et al., 2019), physical reasoning (Bakhtin et al., 2019), mathworld problems (Saxton al., 2019) and general Al (Xie et al., 2021; Chollet, 2019). Some more relevant benchmarks also consider few-shot reasoning setting as ours, including RPM (Barrett al., 2018; Zhang al., 2019; Teney al., 2020), Bongard Problems with synthetic shapes (Nie et al., 2020), physical problems (Weitnauer & Ritter, 2012) and real-world images (Jiang et al., 2022). While the reasoning problems within most of these benchmarks only comprise limited number of simple visual concepts like object categories and shapes, our Bongard-Oper World introduces new challenge of reasoning about free-form open vocabulary concepts in few-shot fashion. Moreover, the open-world nature of our benchmark invites any Al models with no limitation on pretraining, etc, making more friendly to many recent AI advances.
G QUALITATIVE RESULTS
VLM+LLM (single-round). Here we only consider the GPT-4 + BLIP-2 model. The results echo our hypothesis Section 4.2, when BLIP-2 can accurately capture the visual concepts, GPT-4 can perform perfect reasoning (as shown in Figure 4). However, when BLIP-2 misses some details (as shown in Figure 5), it pays more attention to the "bird" in the foreground and ignores the "tree branches covered in snow" in the background, resulting in GPT-4 making correct concept induction but fails on binary prediction.
VLM+LLM (multi-round), Interactive update information can solve this problem instead of using fixed captions, as described in Section 3.3. GPT-4 combines the captions of the query and support images conduct multi-round guestion-answering This allows utilize more detailed and accurate information and perform perfect reasoning, as shown in Figure 6.
Neuro-symbolic approach. We invoke the logic reasoner (detailed Algorithm 2) on GPT-4, which leads to accurate concepts being updated from fixed captions if they are missing, as shown in Figure 7.
worson stage crowd.
BLIP-2
woman in red dress dancing the strett
A woman red dress jumpping in the alr.
A group of women in red dresoes dancing.
The dancing couple on "dancing with the stans"
caption
19
BIJP--2 caption
woman in red dress rumsage with crutches.
A woman in rod suit standing next to wottan zn blue stil..
womnn in red dress is walking down street.
A women dancing black dress stock photo.
ground concept et
woman dencing in red dress.
N
GPT-4 response:
Positiie; wommn real dreys dancing.
Figure 4: GPT-4 correctly produces both binary prediction and induced visual concepts.
22

---

Published as a conference paper at ICLR 2024
BLIP-2 caption
A sasww covered tree in the mudcle of lorest.
A tree brunch covered snow.
A tuee covered snow against blue sky.
tree branch covered Ice and snow.
A rod bard sitting on top of tree beunch.
BLIP-2 caption
The sun shining through the leaves of tree.
The of branch.
bloesom
in
park..
ground truth concept c:
Thee branzhhs covered by snow.
response:
Negatize;
A tree or branch copered in snout.
Figure BLIP-2 only covers unhelpful content Lq. GPT-4 makes correct concept induction fails on binary prediction.
BLIP-2 caption
anow covered tree the middle o' Sonos,
A tree branch coveere in snow.
tree covered in snow against a blue sky.
red hird sitting top ofa troe beunch.
LIM
VIM
P
la
BLIP-a caption
The sun is shining through the laaves of
top
cherry
Q1: Is there snew the
ground truth concept c:
Q2: Are the beanchus snow? A2, Yes.
Tree brunches covered
N
GPT-4 + QA response:
Pasittis; brunches in
Figure 6: Multi-round question-answerrng corrects missing details of images in fixed captions, allowing
GPT-4 to execute correct reasoning.
BLIP-2 eaption
covered in midiile
branch covered
covvered Muee
red hird sitting
P
lg
multi-cund
neuro-symbollc reasoning
BLIP-2 caption
shining through
The of tren branch.
A cherry blnssr Tree in the water.
covcred) Nc.... loavss, tioe. boanch, sonor
Qr nod. hinl siting lees boasci)
ground truth concept
Tiee brunches covered by shoon.
N
updatc Q
sapso Query imagy?
Positise:
GPT-4 logic
snow,
Figure 7: The logical reasoner corrects the missing details of images in fixed captions, enabling GPT-4 perform perfect reasoning.
23

---

Published as a conference paper at ICLR 2024
BLIP=2 caption
brilgg over body water at night.
A view of the brnollye bridlge over water al night.
hridge over river with fireworks the night sky.
bridge over the riverat night with a street light light
bench on bridge at night.
la
standing bridge
walking bridge night.
view of the polden gatee bridge from water.
black and white photo afa bridge over river.
caption
ground truth concept c:
bridge over the water night.
GPT-4V responses
Posihine
Bridges over water bodies.
Figure 8: GPT-4V also experiences hallucinations during reasoning, which leads to difficulties in the conceptual induction of certain images (here is Iq). resulting in failures in binary prediction but remains correct concept induction.
BLIP-2 caption
A snow covered tree in the midile of foresi..
A tree branch covered some.
tree covered in snow agaiinst blue sky.
tree branch covered los and snow.
A Aqquire is leaning ca trie branch.
P
Ia
BLIP-2 cuption
is shining through The top of branch.
cherry blussom tree in the water.
A beach sitting next a tree in park.
ground truth concept
brunches by snowc.
N
Otter response:
Posiriin: branches
Figure 9: The hallucinations in Otter are more pronounced, leading to failures of both binary prediction and induced visual concepts.
BLIP-a captiion
Two womes are enpaged in wrestling match.
Twu wreetlers cumpete on the mat:
Twu wrestlers geaple Curing match.
Two men are arm -wrestlnng motch.
VLM
P
LLMI VLM
msulti-round
acaro-synbolic ressoning
BLIP-e eaption
A grop tunnes compete in road race.
Basketball players shot during
athlete cxptzzed miattoss josdin llmne.
composo, mad, wnarling)
ground truth concept Wrestting competition.
splate Qu..
Qli Are faore wrealens is imag??
Posttior;
(tun).
When concepts require certain level of generalization (which relatively easy humans), concepts extracted from their captions are too chaotic. Any update attempts of Neuro-Symbold logical reasoner fail to yield favorable outcomes, consequently resulting in failures of both binary prediction and induced visual concepts.
Figure 10:
24

---

Published as a conference paper at ICLR 2024
P q
N
Na
(a) With adversarial query selection, the concept is "Red bows".
P
Pa
II
N
Na
(b) Without adversarial query selection, it is straightforward to find similar images in the support set.
Figure 11: Illustration to visually demonstrate ablation analysis design choices.
25

---

Published as a conference paper at ICLR 2024
Table 9: Examples of each category visual concepts in Bongard-OpenWorte We demonstrate the ratio of each concept category more examples, including those mined from CC-3M (anything else) and challenging commonsense-relen concepts that are wowd-sourced and augmented to the dataset.
Concept Category
Ratio
ID
Example
Strawberry leaves.
Kids playing in the river.
Anything else
73.4%
O
A perched mantis hanging a plant.
Reeds swaying in the wind.
Cars the city streets night.
Persons riding bicycles.
Riders riding horses.
HOI
2.7%
People holding the American flag.
A girl is holding a doll.
A farmer wearing hat.
Grilled steaks.
Spicy foods.
Taste Nutrition Food
2.5%
2
A bowl of chocolate pudding.
Some delicious food with shrimp meat.
A plate of vegetable salad.
Steel beams the building.
The tungsten lamp is glowing.
Color Material Shape
5.7%
3
Ceramic bowl.
Leather bags.
Bamboo baskets.
Birds soaring in the air. A colorful balloon floating in the air. The faucet that discharging water. People cheering the soccer field. Small animals that can fly.
Functionality/ Status Affordance
3.0%
The room corner without people. Wedding photos of the bride and groom. A collection of seashells or conches.
And/ / Or Not
1.5%
5
A baboon and car.
A stall selling vegetables and fruits.
Capital of the US.
A distant view of the Egyptian pyramids.
Factual Knowledge
2.8%
6
The American flag.
Golden Gate Bridge.
Sydney Opera House.
Canine animals.
Marine animals swimming in the sea. Military carriers. Ball bats.
Meta Class
2.0%
7
Rescue vehicles.
Coral reefs on the bottom of sea. A feather gently falls on the ground.
Relationship
1.9%
8
A town surrounded by mountains.
An airplane flying above clouds. Books stacked on top of each other.
Clear bottom of lake transmits the sunlight. The moonlight reflected the lake night.
Unusual observations
4.7%
9
A Shimmering water surface. Rusty metal hooks. People in hurry.
26

---

Published as a conference paper at ICLR 2024
H
MORE BONGARD-OPENWORII EXAMPLES
p
N
111
1 19
Figure 12: Example of Bongard-OpenWorld The images in set P all depict "rusty bolts", while the images in set. N have "rusty nails". "rusty wires" or others, so the concept exclusively depicted by "rusty bolls".
27

---

Published as a conference paper at ICLR 2024
p
N
1 1
1 1a2 2
Figure 13: Example of Bongard-Open World. The images in set P. all depict "rubber kayak in the water". while the images set N have "rubber kayak". "boat in the water" others, the concept exclusively depicted by "rubber kayak the water".
28

---

Published as a conference paper at ICLR 2024
p
N
1 1
q 2
Oaee--peo.W....... World. The images in set P all depict "boats docked at the pier", while the images in N have "boats in the ocean". "a boat passing the pier" others, the concept C exclusively depicted by "boats docked the pier".
Figure 14: Example of Bongard-Open
29

---

Published as a conference paper at ICLR 2024
p
N
111 1
1q 2
Figure 15: Example of Songard-Opent World. The images in set P. all depict "person riding bicycle". while the images in N have "person pushing bicycle", "person carrying bicycle" others, the concept C exclusively depicted by "person riding bicycle".
30

---

Published as a conference paper at ICLR 2024
p
N
1 1
q 2
Figure 16: Example of The images in set P. all depict "bamboo baskers". ". while the
images in set N have "plastic baskets". "metal baskers" or others, the concept C' exclusively depicted by P. "bamboo baskets".
31


Reviews:
[{"review_id": "JWLfEit0aU", "reviewer": "Reviewer_bqT5", "summary": "This paper proposes a new benchmark, called Bongard-OpenWorld that contains visual few-shot reasoning tasks. Specifically, a Bongard problem contains a set of ‘positive’ and ‘negative’ images in the support set, where the positives all share a concept that none of the negatives do. The goal is to use this ‘context’ to infer the positive ‘concept’ in order to correctly label a disjoint set of query images as being positive or negative. While this problem has been studied in previous work, the proposed benchmark differs in that the concepts are ‘open world’ (rather than selected from a predefined small set). Specifically, they leverage Conceptual Concepts which is a massive web-crawled dataset containing image descriptions, and extract concepts from that dataset as well as through crowd-sourcing, to obtain concepts that contain factual or commonsense knowledge. Then, an image search tool is used to find appropriate images from the web to populate the ‘positives’ and ‘negatives’ for each concept (as well as query images) in order to form Bongard problems. They conduct an empirical investigation using both canonical few-shot learning methods as well as leveraging LMs and VLMs in different ways. For example, they explore a scenario where the VLM produces a caption for each of the images in the support set, and then these captions along with the positive and negative labels are fed to the LM which makes a prediction for each query image via in-context learning. This can be done in one-go or iteratively. They also propose a symbolic approach that directly applies logical operations to infer the positive concept.", "questions": "- In Table 1, how is the number of tasks computed? What constitutes a unique task? Would having the same set of classes but different images in the support set count as the same task?\n\n- In Fig 3a, different few-shot learning algorithms are shown for the classification head only which seemed surprising. Some of these are meta-learning methods that also update the backbone. Is there a meta-training phase (starting possibly from a pretrained architecture) during which the backbone is also finetuned?\n\n- the authors mention that all few-shot learners excluding ChatGPT and GPT-4 use a ConvNext-base. But they also mention that SNAIL uses a transformer architecture. Should SNAIL be listed as another exception there?\n\n- The authors claim that open vocabulary is important for this benchmark and they use this as a  justification for the fact that pretraining on larger datasets leads to better results (“few-shot learners fueled with proper open-ended pretrained models [...] can alleviate this gap”). But an alternative explanation could be that such large pretrained models like CLIP have already seen the specific images and / or concepts presented in the few-shot learning task and thus they simply face a weaker generalization challenge compared to models that were trained on smaller training set which may have a smaller probability of having seen these exact images or concepts. Have the authors made an attempt to examine or rule out this alternative hypothesis?  \n\n- Is it possible that some of the created Bongard problems are not solvable? E.g. this could happen if there accidentally is more than one concept that is shared between all of the positive images and none of the negative images. Is care taken to avoid this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper studies the interesting problem of few-shot visual reasoning\n- both ‘traditional’ few-shot learning methods as well as newer ideas involving LMs and VLMs are explored\n- the finding that even the best approaches lag significantly behind human performance is an interesting one, and points to the proposed benchmark as a valuable one for pushing the boundaries of of existing methods in this important direction", "weaknesses": "- Some related work is missed. [A] (see references below) studies a setting very related to the proposed benchmark (though they didn’t use the terminology Bongard problems). They also created tasks using natural (‘real-world’) images from different datasets, from using computer vision datasets (rather than scraping the web).\n\n- It would be great to add additional few-shot learning baselines to cover families of approaches that are excluded from the current analysis like approaches that perform FiLM-conditioning e.g. [B, C] (see references below) and approaches that train the backbone with gradient descent within each task, like MAML and Proto-MAML (the latter is a version proposed in the Meta-Dataset paper which is cited by this work)\n\n- The paper has some clarity issues, perhaps owing to the fact that the authors tried to ‘squeeze’ a lot of content in the required number of pages. It’s hard to fully understand the different methods by reading only the main paper. I found the neuro-symbolic method proposed especially hard to understand (even after looking at the algorithm in the appendix). Please include some higher-level motivation and the intuition for the particular updates that it entails.\n\n- In Table 2, it’s hard to tell which methods / rows correspond to which of the families of approaches (e.g. a, b, c, or d in Figure 3) – and e.g. which are single-round or multi-round. Perhaps a good way of indicating this is by adding an extra column in that table.\n\n- It would be great to conduct ablation analyses for design choices made in creating the benchmark, like the adversarial query selection that picks the positive query to not be too close to the prototype of the positive class. \n\n- It would be great to conduct an analysis of the effect of the ‘shot’ on these problems. Intuitively, the more positive and negative images the network sees, the easier it is to infer what is the positive class and correctly label query images. Given the negative results in the paper with the current number of shots (6 positives and 6 negatives), in terms of the very large gap from human performance, have the authors considered increasing the number of shots? Understanding how performance of different methods differs as the number of shots increases would be insightful.\n\n- it would also strengthen the paper to tie in the findings of this work with findings in related works. E.g. in the Bongard-HOI benchmark that the authors claim is the most similar, do they have similar findings e.g. in terms of which methods perform better?\n\n\nMinor\n=====\n- ‘given 6 positive and 6 negative images [...] (see Figure 1 for illustration)’ – but Figure 1 shows only 3 positive and 3 negative images (6 in total, not each). Maybe clarify that Figure 1 doesn’t correspond to that setting and is used for illustration only? Or describe the task in the intro at a higher level of abstraction, e.g. P positive and N negative images.\n- in the caption of Figure 1, highlight ‘hard negatives’ in orange, like ‘distractors’ are highlighted in green, to match the (captions of the) images shown in that figure.\n- typo: “prob” → “probe” (on page 6)\n- typo: “was not fine-tuning” → “was not fine-tuned” (in Table 2’s caption)\n\nReferences\n=========\n\n- [A] Probing Few-Shot Generalization with Attributes. Ren et al.\n\n- [B] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes. Requeima et al. NeurIPS 2019.\n\n- [C] Improved Few-Shot Visual Classification. Bateni et al. 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qWQuoNSuNe", "reviewer": "Reviewer_yoip", "summary": "The authors claim that they proposed Bongard-OpenWorld, a new benchmark for evaluating real-world few-shot reasoning for machine vision. Based on this benchmark, they further present the few-shot learning baseline approach.", "questions": "Please refer to Weakness.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors claim that they proposed Bongard-OpenWorld, a new benchmark for evaluating real-world few-shot reasoning for machine vision. Based on this benchmark, they further present the few-shot learning baseline approach.", "weaknesses": "1. In the experiments, the authors primarily focus on conducting investigations using real-world datasets, particularly the their self-constructed dataset. However, given the Bongard Problem, it raises concerns about the generalizability of the conclusions/findings obtained from real-world datasets to mathematical datasets.\n\n2. The experimental results seems to ignore the traditional models, and it remains a concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zNzO5zAVR9", "reviewer": "Reviewer_CSkd", "summary": "The paper introduces a new benchmark called Bongard-OpenWorld, which focuses on open-world visual concept understanding. The task is to classify a query image to belong to one of two sets of images. The positive set contains 6 images depicting a common concept C, such as \"animals are running\". The negative set contains 6 images of similar concepts but not exactly matching C, e.g., showing a standing animal, or a running robot. The difficulty of this benchmark comes from the two sets sharing common objects or semantics, such that nuances in the semantic concepts need to be understood to perform well. The authors also evaluate relevant existing methods and show that there is still a large gap between current methods and human performance.", "questions": "- Have you thought about not providing the labels \"positive\" and \"negative\" for the two sets to the methods? Why have you chosen this setup?\n- In Sec 2.1: What is a grid in this context? What is grid sampling? How do you define \"concept tuples\"? Neither the main paper, nor the supplementary clarifies this sufficiently.\n- How do you ensure that the dataset does not contain duplicate concepts? I assume this is the case because in Table 1, it is reported that the dataset has 1.01K concepts and 1.01K tasks.\n- What are the exact instructions the annotators were given? For instance, when \"annotators are instructed to write visual concepts by following a predefined set of categories illustrated in Table 7\" and when \"they are also asked to combine these challenging concepts with those mined from CC-3M\" (Sec. 2.1).\n- Images are collected by using an online search based on the concepts. What is the license of the images collected? Do the authors have the rights to distribute the images?\n- In Sec. 2.2. you write: \"the annotators are then asked to provide two sets of candidates for positive and negative queries\". How many images are collected here for possible query images? Why choose only one positive and negative image as query in the end?\n- Does defining the concepts of category 0 (from CC3M) undergo any crowd-sourcing or is it fully automated?\n- Is there performance difference between positive and negative queries?\n- In Figure 2c, both x and y-axis should be labeled. What is the scale/size of the number of concepts (x-axis)? What is the unit of the numbers on the y-axis?\n- What is meant by \"we report the overall accuracy of all models\". Does Table 2 report test set accuracy or accuracy over the whole dataset, i.e., including training samples?\n- Why are concepts from CC3M considered non-commonsense?\n- How is ChatGPT finetuned (Table 2)? Does this use the finetuning API of OpenAI? More details would help make this more reproducible.\n\nComments/suggestions:\n- Table 2 includes methods that use training data to update NN weights and others that do not update weights (\"zero-shot\" setting). It would be much clearer if the table indicates which models use training data.\n- The following phrase appears 3 times in the manuscript. I suggest to to reduce this repetition and rephrase it according to the context. \"We even designed a neuro-symbolic reasoning approach that reconciles LLMs & VLMs with logical reasoning to emulate the human problem-solving process for Bongard problems\".\n- Sec. 3.2 (at the end) promises captioning metrics, but they do not appear in the main paper, only in the supplementary.\n- The formatting of Table 7 is confusing. It would be better to clearly separate the left half from the right half, or simply just make it 10 rows.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The new benchmark and problem setting tackles an important shortcoming of current methods to understand fine-grained semantic concepts in contrast to hard negatives.\n- An extensive set of existing methods have been evaluated showing that the best models (64%) are still far from human performance (91%). This makes it a challenging setting for new methods to be developed in the future.\n- Evaluations include a few-shot and zero-shot setting, and several different approaches to combine vision-language models and large language models to solve the task.", "weaknesses": "- The presentation and writing could be clearer, making some parts of the paper difficult to understand, especially around the creation of the dataset in Sec. 2 and the precise problem setting. For example, it was not immediately clear that the labels of the two sets (positive, negative) are given and do not need to be inferred. The query image can belong to either, but the name \"positive\" suggests that this is the GT set of the query image, which is not the case. I am adding more clarifying questions in the questions section below.\n- The problem setting is imbalanced. The positive set corresponds to a single concept while the negative set does not, but instead contains a subset of the complete concept. While this is not necessarily an issue and can be a design choice, there is no justification why this choice has been made. For instance, why not have both sets correspond to a single concept where the contrasting sets are close in semantics to make it a hard problem? Similarly to how different splits are evaluated in Table 2, it would have helped to show the performance of positive query images vs. negative query images in order to understand if this imbalance makes positive/negative queries easier/harder.\n- While a lot of models have been evaluated on the proposed benchmark, a natural baseline is missing: computing the image similarity between the query image and the two sets. For instance, one can use any pre-trained image encoder (CLIP, Dino, etc.) or image-to-image retrieval method and use the mean similarly of the image embeddings per set to make a prediction. Using captioning models and LLMs seems to introduce complexity while at the same time discarding fine-grained image information by only relying on text to make the decision.\n- With around 1K tasks it is a rather small dataset. Hence, focusing on the \"zero-shot\" setting without involving training might be the better use case.\n- While it is true that the benchmark contains a large variety of concepts, positioning it heavily as an \"open-world\" and \"open-vocabulary\" task could be a bit misleading as the core problem is to identify whether an image came from set A or set B. The optional task of naming the concept is most fitting for \"open-world\", but it serves a minor role in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O1gepjsdND", "reviewer": "Reviewer_Gtsv", "summary": "The paper introduces Bongard-OpenWorld, a benchmark designed to evaluate a system's real-world few-shot reasoning abilities. By incorporating open-world concepts and real images into the classical Bongard Problems, this benchmark serves as a litmus test for current limitations in visual intelligence, motivating further research toward enhancing few-shot reasoning in visual agents. The paper conducts a comprehensive assessment, examining the effectiveness of various Vision-Language Models (VLMs) and Large Language Models (LLMs), as well as proposing a neuro-symbolic reasoning approach tailored for this benchmark.", "questions": "Table 2 provides a good overview of the performance of various approaches on the proposed benchmark. I would like to see more explanation of the reasoning behind the performance of these approaches. Like for example, Flamingo/ChatGPT/Otter performs significantly worse than the few-shot learning approach SNAIL despite Flamingo/Otter using the same image encoder. \n\nIncluding a section on failure case analysis for different approaches would be instrumental for the readers in identifying specific challenges and guiding improvements for tackling them.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall the paper is well written and provides the reader with good insight on why the availability of the proposed benchmark is good for the community as it promotes research into the few-shot reasoning capabilities of current black box deep learning models. The paper presents a tough benchmark that tests current systems on their ability to reason about free-form concepts in a few-shot manner by identifying the common concept in a positive set and distinguishing it from the negative set. \n\nThe paper introduces a robust benchmark that assesses systems' ability to perform few-shot reasoning on free-form concepts. It challenges models to identify commonalities in positive sets while distinguishing them from negative sets, enhanced by the inclusion of distractors and hard negatives in the dataset curation process. The evaluation framework covers a wide spectrum, encompassing four distinct approaches: few-shot learning, combined LLM+VLM in single and multiple steps, and a novel neuro-symbolic architecture.\n\nThe evaluation setup utilized in the paper is comprehensive and includes the evaluation of four different kinds of approaches that include a few shot learning approaches, LLM+VLM in a single step, LLM+VLM in multiple iteration steps, and finally a proposed neuro-symbolic architecture.", "weaknesses": "I would like to look at more variants of the neurosymbolic approach proposed in this work. One avenue worth exploring is a line of research that leverages domain knowledge, such as knowledge graphs, to identify pertinent concepts within an input. Active nodes within the graph could then be employed to pinpoint the common concept within the positive set of images.\n\nThe evaluations used in this paper though really comprehensive, miss out on some more ways of evaluation. VLM-based approaches, like GPT4(V), that directly take images as input and can be prompted to obtain the desired input, could be used to identify the relevant concept from a collage of images given together. Since current VLM/LLM approaches are very susceptible to the way they are prompted, it is very important to prompt engineer them in a number of ways and then identify the best working one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)