BloomVQA: Assessing Hierarchical Multi-modal Comprehension (2024)

Abstract: We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus on fact-based memorization and simple reasoning tasks without theoretical grounding, we collect multiple-choice samples based on picture stories that reflect different levels of comprehension, as laid out in Bloom’s Taxonomy, a classic framework for learning assessment widely adopted in education research. Our data maps to a novel hierarchical graph representation which enables automatic data augmentation and novel measures characterizing model consistency. We perform graded evaluation and reliability analysis on recent multi-modal models. In comparison to lowlevel tasks, we observe decreased performance on tasks requiring advanced comprehension and cognitive skills with up to 38.0% drop in VQA accuracy. In comparison to earlier models, GPT-4V demonstrates improved accuracy over all comprehension levels and shows a tendency of bypassing visual inputs especially for higher-level tasks. Current models also show consistency patterns misaligned with human comprehension in various scenarios, demonstrating the need for improvement based on theoretically-grounded criteria. The dataset can be accessed at https://huggingface. co/datasets/ygong/BloomVQA.

heading: 1 Introduction
text: Recent advances of machine intelligence solutions have demonstrated tremendous success in a wide range of language and multi-modal tasks over diverse domains (Bommasani et al., 2021; Brown et al., 2020; OpenAI, 2023; Liu et al., 2023b; Touvron et al., 2023). With increasing popularity of such solutions, how to systematically evaluate and
∗: These authors contributed equally to this work †: work completed at SRI International ‡: work completed at Rochester Institute of Technology
analyze the models remains a key concern for confident application. Many recent efforts have focused on probing the capabilities and risks of the models based on different tasks, measures and perspectives (Chang et al., 2023; Li et al., 2023a; Sawada et al., 2023; Wang et al., 2023). For example, recent studies have explored functionalities and limitations of large language models (LLMs) from the perspectives of cognitive science (Mahowald et al., 2023) and semantic consistency (Sahu et al., 2022; Schiappa et al., 2023).
In the context of Visual Question Answering (VQA), several benchmarks target demonstration of multi-modal reasoning and comprehension beyond mere memorization of low-level statistical patterns in the data. For example, GQA (Hudson and Manning, 2019) leverages scene graph representations to construct VQA tasks describing reasoning based on relations of different entities in single-frame images. Other datasets (Tapaswi et al., 2016; Yagcioglu et al., 2018) leverage sequential visual inputs to form evaluations addressing procedural tasks or story comprehension over longer time horizons. However, little effort has been made in curating datasets to support systematic evalua-
14905
tion of multi-modal comprehension following a theoretically-grounded definition of comprehension, leading to tasks describing only a limited set of cognitive processes at relatively low levels. Motivated by this challenge, we draw inspiration from theories and methodologies developed by education researchers on K-12 reading comprehension, where comprehension tasks are graded based on underlying cognitive processes involved following a taxonomy named Bloom’s Taxonomy (Bloom et al., 1956; Anderson et al., 2001).
In Bloom’s Taxonomy (Figure 2), handling of knowledge is categorized into 6 levels from the most basic (Level 1) to the most advanced (Level 6), including “remember,” “understand,” “apply,” “analyze,” “evaluate” and “create.” For each level, a group of cognitive skills are identified as required to handle tasks at the corresponding level. For example, the skill of differentiating is required to achieve successful analysis (Level 4), while the skill of critiquing is required to achieve successful evaluation (Level 5). Following this taxonomy, we propose BloomVQA dataset containing VQA tasks categorized based on the level of cognitive skills required. To support a wide range of tasks reflecting basic and advanced comprehension, we collect multiple-choice VQA samples with open-ended questions and free-form answer choices based on picture stories designed for early childhood education. Serving as an evaluation set for multimodal comprehension, BloomVQA contains 1200 core data samples collected from human annotators based on 20 stories, categorized into 6 Bloom’s levels. The core data samples are further augmented based on the hierarchical structure of Bloom’s Taxonomy. Specifically, we generalize the concept of scene graphs and propose a novel graph representation named the Story Graph (Figure 1). While
scene graphs focus on attributes and relations of low-level entities mapped to knowledge mostly at Level 1 in Bloom’s Taxonomy, to accommodate advanced tasks at higher Bloom’s levels, we consider different events, each of which can presumably be described by a separate scene graph, as the nodes of a Story Graph. We consider edges describing relations between pairs of events. In addition to low-level relations (e.g., temporal relations), we describe relations corresponding to underlying cognitive processes (e.g., making inference, making prediction) which link different events in reaching an overall comprehension of the story. In this way, given a VQA task constructed about one event, new VQA tasks can be automatically constructed at a combinatorial scale by traversing the underlying Story Graph, as information recorded along the path of traversal can be introduced as the context in forming new VQA tasks.
BloomVQA can be used as a benchmark for multi-modal comprehension engines including current multi-modal LLMs (OpenAI, 2023; Liu et al., 2023a; Zhu et al., 2023), as it enables systematic evaluation of model capabilities with quantitative metrics characterizing its alignment to human comprehension, which is an important aspect for models to approach Artificial General Intelligence (AGI) (Morris et al., 2023). First of all, with data categorized based on Bloom’s levels, we can probe models in a graded manner and examine model behaviors on tasks posing different levels of challenges for human comprehension, especially including tasks requiring high-level cognitive skills. Furthermore, we propose novel metrics characterizing the consistency of model performance following the underlying hierarchy and examine intuitive hypotheses mapping model behaviors to human comprehension patterns. In this work, we examine
the hypothesis that models succeed at more challenging tasks are less likely to fail at easy ones. We also study how relevant knowledge, where the relevancy can be defined based on the underlying taxonomy, would affect model performance. For example, lower-level knowledge which can be comprehended using basic cognitive skills (e.g., memorizing "what is the name of the girl?") is unlikely to have a big effect on solving tasks that require more advanced cognitive skills (e.g., making inference about "what would happen if the girl is taking a flight rather than a train?"). In addition to being accurate, being consistent and consequently predictable is critical for models to be applied by human users with confidence.
Using the proposed dataset and metrics, we conduct comprehensive evaluation of recent VisionLanguage Pre-training (VLP) models (Radford et al., 2021; Li et al., 2022b, 2023b) and a stateof-the-art multi-modal LLM (OpenAI, 2023). We observe that for all three VLP models, the comprehension accuracy degrades for tasks requiring higher-level cognitive skills, in comparison to tasks requiring mere memorization of low-level details that can be directly identified from either textual or visual data. With zero-shot prompting experiments using GPT-4V (OpenAI, 2023), we observe generally higher VQA accuracy over all levels. However, there is also an increasing tendency for the model to take shortcuts and bypass visual comprehension when considering tasks requiring higher-level comprehension, as shown in Figure 2. Furthermore, via consistency analysis, we observe that model behaviors deviate from human comprehension patterns and intuitions at various scenarios. The results demonstrate the need for examining and improving current solutions based on theoretically-grounded criteria of reasoning and comprehension. In summary, our core contributions include the following:
• We present a novel BloomVQA dataset for systematic evaluation of multi-modal comprehension. The dataset is constructed based on Bloom’s Taxonomy, a classic framework from education research which provides categorization of learning and comprehension.
• We propose the Story Graph, a novel hierarchical graph representation based on Bloom’s Taxonomy for story comprehension which enables meaningful data augmentation via traversing the graph.
• We propose novel metrics based on the underlying data hierarchy to examine the consistency of model performance for multi-faceted analysis on model reliability.
• We evaluate state-of-the-art models with our proposed dataset and metrics. We show that improvement of current solutions is needed especially to address tasks reflecting higherlevel cognitive skills and to demonstrate consistent and reliable comprehension.

heading: 2 Related Works
text: Bloom’s Taxonomy First established by Bloom et al. (1956) and further revised by Anderson et al. (2001), Bloom’s Taxonomy describes a framework categorizing learning objectives based on levels of complexity of underlying cognitive processes. There are six major categories, each represented by a set of skills and actions describing the corresponding cognitive processes. Specifically, direct recall of concrete facts and knowledge is considered as the basis (Level 1) of objectives requiring higher-order reasoning and abstraction, from understanding and being able to explain (Level 2) to creating and generating original works (Level 6). While Bloom’s Taxonomy has been widely adopted in K-12 and college education as the guidance for designing teaching instructions and assessments (Thompson et al., 2008; Shuhidan et al., 2009), limited effort has been made to transfer the taxonomy to the machine learning domain as the guidance for analyzing, assessing and improving learning. Zhang et al. (2021) propose a learningbased model for automatic classification of education questions based on Bloom’s Taxonomy. Sahu et al. (2021) use Bloom’s Taxonomy to form proximal clarifying contexts and study their impact on language models. In this work, we propose a novel VQA dataset to support hierarchical model assessment guided by Bloom’s Taxonomy. Specifically, we use the action verbs associated with different Bloom’s levels to create templates for data collection. While it is noted in recent studies that using the action verbs alone may not be sufficient in fully describing the underlying learning process (Larsen et al., 2022), the verbs provide us a concrete operational definition of the taxonomy as the guidance for understanding a coarse trend of model performance over data categorization.
Visual Comprehension and Reasoning Motivated by the multi-modal nature of human comprehension and communication (Alikhani et al., 2023), many recent datasets seek to challenge VQA models by tasks requiring reasoning in different domains (Zakari et al., 2022; Garg et al., 2022; Kafle et al., 2018, 2020). Several works introduce tasks requiring understanding based on multiple images. MovieQA (Tapaswi et al., 2016) dataset contains large scale multiple-choice samples collected for story comprehension based on video clips. RecipeQA (Yagcioglu et al., 2018) is constructed based on recipes with procedural text and image instructions to evaluate visual understanding over events with temporal relations in the form of clozing and ordering tasks. SlideVQA (Tanaka et al., 2023) proposes data and tasks requiring multi-hop reasoning over slide decks with multiple pages. GQA (Hudson and Manning, 2019) leverages Visual Genome annotations (Krishna et al., 2017) to introduce tasks requiring reasoning with respect to entity relations in scene graphs.
In comparison to existing data where categorization is either based on the prefix of the question or intuitive summary of the tasks, we provide principled and theoretically-grounded categorization based on Bloom’s Taxonomy. Furthermore, we generalize the concept of scene graphs and propose the Story Graph which integrates entity-level relations, temporal relations and relations defined by the cognitive processes from Bloom’s Taxonomy, enabling hierarchical data augmentation and comprehensive model evaluation with novel metrics.

heading: 3 BloomVQA Dataset
text: Data Collection We propose a novel dataset for systematic assessment of multi-modal comprehension. We collect picture stories designed for educating young children from two Creative Commons (CC BY 4.0) resources (StoryWeaver; Book Dash) which provide collections emphasizing an Indian and an African cultural background respectively. We manually select 20 stories with relatively consistent artistic style, length (around 10-20 pages) and plot complexity so that tasks reflecting different levels of human comprehension and requiring different types of cognitive skills can be meaningfully collected based on the content of the stories. We filter out the stories with distorted artistic styles as they can pose additional challenges to both human annotators and learning models.
We provide a web-based UI (details included in the Appendix) where an annotator is asked to read through a picture story and then provide a set of 6 multiple-choice samples in English, each corresponding to one level in Bloom’s Taxonomy. In Bloom’s Taxonomy, each level of comprehension is associated with a set of action verbs describing underlying cognitive processes and skills (e.g., "identify" for Level 1). We use the action verbs to construct a set of more than 70 question templates as our operational definition of Bloom’s Taxonomy. For example, "How would you compare the <attribute> of <character> at the beginning and the end of the story?" is provided as a template based on the Level 4 action verb "compare". More examples are included in the Appendix. For each sample, we collect one template-based question and 4 free-form answers including 1 correct answer and 3 incorrect answers from the same annotator.
We collect data samples via Amazon Mechanical Turk (AMT) and manually review all inputs to select data with proper reflection of the story and the underlying Bloom’s level. For each story, we collect 10 sets of reviewed inputs from different annotators. To further reduce the bias in the dataset, we enforce annotators to provide answer choices with consistent lengths for the same question, with a length difference fewer than 5 words.
Detailed statistics about the data are shown in Table 1. In contrast to VQA data considering simple answers, we collect data with long freeform answers especially for high-level comprehension tasks. Overall we collect a core set of 1200 multiple-choice samples, based on which we perform systematic data augmentation as described in Sec 4. We demonstrate consistency analysis with a set of 12k augmented samples.
We perform human baseline on proposed dataset (1200 core VQA samples) with a small group of adult reviewers. The human baseline reports an average accuracy of 89% with 2% standard deviation.
This preliminary study provides insights on the current gap between human and machine capabilities (GPT-4V reports an average accuracy of 75.3% as shown in Table 2) on the proposed dataset.
Story Graph Grounding VQA data to an established taxonomy provides not only theory-based categorization, but also guidance on systematic probing and generation of the data. Based on Bloom’s Taxonomy, we propose a hierarchical graph representation for picture story data. Referred to as the Story Graph, this novel representation naturally extends the concept of scene graphs (Krishna et al., 2017). While scene graphs focus on representing low-level information such as geometric relations between local entities based on individual images, in a Story Graph (Figure 1), we represent different events in a single story as the nodes and represent event-level relations as the edges. In addition to low-level relations (e.g., temporal relations), we include a wide range of higher-level relations (e.g., logical relations, causal relations) corresponding to different levels of cognitive skills specified in Bloom’s Taxonomy (e.g., making comparison, making inference). As shown in Fig. 1, each of our templates correspond to one type of the edges in the Story Graph describing a particular cognitive skill. The type of edge is the same for all instantiations of the template. For example, the template "<Character><Action>, what would you do to change this action? <Answer>" always creates a Level 3 edge between two events, each encoding Level 1 details such as the character involved and the action taken. In this way, rich and diverse knowledge about the story requiring different levels of comprehension is organized in a hierarchical manner. The benefits of the proposed Story Graph are multi-fold. By traversing through the underlying graph, we can achieve systematic augmentation of VQA data following the taxonomy encoded in the graph. For example, given a task about a base event, we can construct an augmented question by incorporating knowledge about each connected event as the context. This augmentation not only expands the dataset combinatorially, but also introduces novel metrics for assessing the consistency of learning models. As context information introduced in the augmentation is labeled with Bloom’s levels, consistency analysis on how context knowledge requiring different levels of comprehension would affect base tasks can be performed to characterize model reliability.

heading: 4 Experimental Analysis
text: To demonstrate the multi-faceted assessment enabled by our framework, we perform zero-shot experiments using three recent VLP models including CLIP (Radford et al., 2021), BLIP (Li et al., 2022b) and BLIP2 (Li et al., 2023b) with reported implementations (Li et al., 2022a). We further perform zero-shot prompting-based experiments using the state-of-the-art GPT-4V (OpenAI, 2023). The baseline models are proposed to demonstrate a wide range of vision-language tasks probing multimodal comprehension. For CLIP experiments, we use pretrained ViT-B-32 image transformer. For BLIP and BLIP2 experiments, we use pretrained models in the image-text matching (ITM) mode (Li et al., 2023b). The models contain 583M and 188M trainable parameters as reported in the original papers. We use one Nvidia Tesla V100 GPU for experiments with VLP models.
Multiple-choice Accuracy We first evaluate the models with our core dataset of 1200 multiplechoice samples under two settings. In the first setting, we perform the "Hasty Student" experiment (Tapaswi et al., 2016) by predicting multiplechoice answers solely based on the question text. This baseline measures underlying biases, as high accuracy indicates that the model is making up answers effectively without reading the actual picture stories. For experiments with three VLP models, we extract text embeddings of the question q and each answer candidate ai from the set of answer choices {ai}N=4i=1 using the text encoder and examine cosine similarity between the question embedding and each candidate answer embedding. We select the answer corresponding to the largest QA similarity as our prediction.
In the second setting, we perform the "Searching Student" experiment (Tapaswi et al., 2016) by predicting the multiple-choice answer based on the cross-modal similarity defined as
l({vj}Kj=1, q, ai) = max j (g(q, vj) + g(ai, vj))
where g denotes the cosine similarity between normalized embeddings extracted from the text input (question q and answer candidate ai) and the image frames in the picture story where {vj}Kj=1 denotes a set of image frames from a story with K pages and vj denotes the jth frame. We adopt max pooling for aggregating the similarity scores over image frames based on empirical performance
of baselines. Other aggregation methods such as average pooling are tested with only minor performance differences noted. We select ai⋆ with i⋆ = argmaxi l({vj}Kj=1, q, ai) as the model prediction.
Model Consistency: Conditional Performance Categorizing tasks based on Bloom’s Taxonomy enables evaluation of the consistency of model performance over tasks at different levels. As for humans, being consistent on relevant tasks is an important sign of comprehension on given subjects. In this work, we examine an intuitive hypothesis: models which succeed at more challenging tasks demonstrating comprehensive cognitive skills should be less likely to fail at easier ones requiring only basic skills. Let Xm and Xn denote a pair of VQA tasks from the same story and the same annotator at Bloom’s levels m and n respectively. We compute the likelihood of having a model solving Xm correctly given that the model is correct on Xn as
Pm,n = 1
|S|
|S|∑
s=1
1{âm,s = ām,s|ân,s = ān,s},
where |S| denote the total number of annotation sets. â and ā denote the predicted answer and correct answer for a task respectively. If Pm,n with m < n is higher than the unconditional accuracy at level m, it suggests that the model performance complies with the intuitive hypothesis. In other words, for a model demonstrating consistent comprehension patterns resembling human behavior, its probability of solving easier tasks from a lower Bloom’s level m is expected to be high when the model is correct on more comprehensive tasks from a higher Bloom’s level n. Note that the formulation of consistency as conditional performance can be further expanded based on different hypotheses to examine various forward and backward consistency patterns (Chen et al., 2023).
Model Consistency: Augmentation Based on the data augmentation strategy described in Sec. 3, we propose another strategy for probing model consistency by comparing its performance on visual comprehension tasks with and without background knowledge introduced. We consider the relevancy of background knowledge to a given task to be associated with the node connection in an underlying Story Graph constructed following Bloom’s Taxonomy. For a consistent model, its likelihood of success on a task should only be improved when
relevant background information is provided and should not be affected by irrelevant information. Correspondingly, limited information from a lowerlevel task is less likely to affect the model performance on a higher-level task requiring comprehensive understanding of the story. While background information from a higher-level task involving advanced cognitive skills and reasoning is more likely to be useful for inferring the answer of a lower-level task. For example, considering a Level 3 base task ("If you are Nina’s father, what would you do to change her mood?") which requires distilling information from the story for problem solving, background information from a Level 1 context task ("What’s Nina’s father’s hair color? Black.") is less likely to be helpful. Therefore, a consistent model would have a similar performance on the augmented task ("If you are Nina’s father, who has black hair, what would you do to change her mood?") given the same set of candidate answers. Meanwhile, the background information from a Level 4 context task ("Why is Nina sad? Because she misses friends.") is expected to facilitate comprehension, leading to improved performance on the augmented task ("Nina is sad because she misses friends. If you are Nina’s father, what would you do to change her mood?").
Given a context task Xn and a base task Xm drawn from the core dataset at level n and m respectively, we construct an augmented task Xm|n by prepending the question qn and the correct answer ān of the context task to the base question qm. We keep answer choices of the base task am. For experiments with three VLP models, we extend the Searching Student formulation for the augmented task by defining the cross-modal matching score between each candidate answer choice am,i and the picture story with frames {vj}Kj=1 as
l({vj}Kj=1, qn, ān, qm, am,i) = maxj(g(qn, vj) +g(ān, vj) + g(qm, vj) + g(am,i, vj)).
We select the answer corresponding to the highest score as the prediction for the augmented task. While combinatorial augmentation can be achieved, in this work, we examine a set of 12k augmented samples considering Level 1 data of the same story as the context task. Let Ybase denote the model performance as a binary score for each base task without augmentation. Let Yaug denote the model performance as the average accuracy on the set of augmented tasks constructed by incorporating
low-level contexts. We compute consistency defined as the average precision between two sets of scores (Sahu et al., 2022): AP (Ybase, Yaug). This metric quantifies the predictability of model performance when probing the knowledge along the underlying hierarchical graph.
Experiments using GPT-4V We perform zero-shot prompting experiments using the gpt-4-1106-preview model and the gpt-4-vision-preview model. In comparison to experiments using VLP models, we ask GPT-4V model to make a prediction based on all images in each story at once. We perform experiments on the core dataset with 1200 samples and a set of 1200 augmented data constructed by considering a random Level 1 context task from the same story for each base sample from the core dataset. We perform computation via OpenAI API. The total computation corresponds to approximately 8M tokens. We use the following prompts for multiple-choice evaluation. For each task, we randomly order the candidate answers and parse the output of LLM to compare the index generated following “My chosen answer is” to the ground-truth index of the correct answer. For GPT-4 experiments, we use the prompt:
Choose the best answer based on the question. End your response with ‘My chosen answer is’ followed by your chosen answer. <Question> <Candidate Answers>
For GPT-4V experiments, we use the prompt:
<Images> Choose the best answer based on the story in the images. End your response with ‘My chosen answer is’ followed by your chosen answer. <Question> <Candidate Answers>
With prompts specified above, we receive indeterminate answers for 5% and 2% of the samples in QA-only and VQA experiments respectively. We report the average performance on the rest of the samples in Table 2-4.

heading: 5 Results and Discussion
text: Multiple-choice Accuracy In Table 2, we present graded evaluation on BloomVQA dataset with three recent VLP models (Radford et al., 2021; Li et al., 2022b, 2023b) and a latest multi-modal LLM (OpenAI, 2023). We first compare the model performance on the core set of 1200 samples considering text inputs only ("QA") and multi-modal inputs ("VQA"). It is shown that all three VLP models have consistently low performance over different Bloom’s levels when making prediction based on QA similarity solely. In fact, the average performance is close to the random guess which demonstrates that the proposed dataset serves as a proper evaluation set for multi-modal data. In comparison to the QA baseline, all three VLP models demonstrate improved performance when exploiting multi-modal similarity for VQA prediction for most Bloom’s levels. We have similar observation for all three VLP models such that they achieve higher accuracy for multi-modal comprehension on lower-level tasks (level 1-2) in comparison to higher-level tasks (level 3-6). For comparison between three VLP models, we observe that the recent improved model (BLIP2) achieves the most gain in low-level (Level 1) tasks while similar performance is achieved by different models for higher-level tasks. This observation supports our hypothesis that VQA tasks corresponding to high-level human comprehension are not adequately addressed by current techniques, as existing datasets and models have been focusing on tasks reflecting memorization of low-level knowledge.
Model Consistency: Conditional Performance In Table 3, we consider the 1200 core samples and report model performance on tasks of a specific Bloom’s level (m) conditional on correct prediction on the task of the same story from the same annotator at a different Bloom’s level (n). With this set of results, we seek to compare the model performance with human comprehension patterns.
We start with the hypothesis that comprehension at a lower level is less likely to fail when comprehension at a higher level is already achieved, as many lower-level comprehension skills are readily required in solving higher-level tasks. We observe a pattern in model performance aligned with this hypothesis in some scenarios. For example, as shown in Table 3a, Level 2 accuracy given success of higher-level (Level 3-6) tasks are generally higher than Level 2 accuracy given success of lower-level (Level 1) tasks. However, this observation does not hold across different Bloom’s levels and different models. For example, as shown in Table 3c, Level 5 accuracy conditioned on success of higher-level (Level 6) tasks is lower than Level 5 accuracy conditional on success of lower-level tasks (Level 1-4). The misalignment between model consistency patterns and human comprehension patterns surfaces the lack of grounding in model responses.
Model Consistency: Augmentation In Table 4, we further compare the performance of different models evaluated on VQA data samples with and without augmentation. For core data samples at
each Bloom’s level, we augment the data by incorporating Level 1 knowledge about the same story as the context for the original question ("aug"). We further quantify the consistency of model performance between data with and without augmentation using Average Precision ("AP") scores as described in Section 4. Overall we observe that all three VLP models have a similar consistency level averaged over 6 different Bloom’s categories. The high consistency scores indicate that model performance can be perceived with confidence as lowlevel distractions have small effect on the model performance. Considering consistency measured with base tasks at different Bloom’s levels, we observe that Level 3 and Level 4 base tasks tend to be affected at a relatively greater scale by additional context from Level 1. This observation serves an example of model-specific comprehension patterns disclosed by proposed consistency analysis.
Experimental Results using GPT-4V As shown in Table 2, GPT-4V shows strong performance across different Bloom’s levels. However, the comparison between text-only experiments using GPT4 ("QA") and multi-modal experiments using full GPT-4V ("VQA") indicates that the high accuracy of the model may stem from biases and hallucinations. The model’s gain from incorporating visual inputs decreases on tasks with increasing Bloom’s levels, suggesting a tendency of the model to take shortcuts especially for tasks requiring higher-level comprehension. Specifically at Level 6, the model performance degrades when visual data is introduced. In comparison to VLP models, GPT-4V handles visual inputs at story level. This may con-
tribute to the difference in model performance. We choose picture stories designed for young children as they have simple plots suitable for demonstrating various comprehension tasks without introducing unnecessary complexity. However, with such data there can potentially exist common-sense biases especially for tasks at higher levels which require more abstract comprehension, as the stories are often based on common sense. Therefore the higher performance of GPT-4V in comparison to other baselines can also be a sign that the model is better at leveraging common sense. In this work, instead of comparing models with respect to absolute accuracy, we focus on examine whether the model prediction is grounded to the visual data and the consistency patterns of human comprehension.
In Table 3d, we show GPT-4V performance conditional on successful prediction on tasks about the same story at a different Bloom’s level. We observe that the model consistency pattern deviates from the hypothesis made based on human comprehension patterns in multiple cases. For example, Level 5 accuracy conditional on the success on higherlevel (Level 6) tasks is lower than Level 5 accuracy conditional on the success on lower-level tasks (Level 1-4). Furthermore, as shown in Table 4, in comparison to the VLP models, GPT-4V has a lower average precision score which corresponds to lower consistency on data with and without augmentation by introducing irrelevant information. These observations suggest that, although GPT-4V is demonstrating generally improved VQA accuracy in comparison to earlier models, it may still fall short in demonstrating consistent comprehension following human intuitions.

heading: 6 Limitations
text: Constrained by the resources and availability of creative commons data, we collect a relatively small-scale dataset to serve as an evaluation set for characterization of model performance based on a theoretically-grounded taxonomy. While domain gaps may exist when evaluating models pre-trained with data from different domains, the focus of this work is not to optimize VQA accuracy in the picture story domain. Instead, we focus on providing insights on model behaviors over different types of comprehension and measuring its consistency. Our proposed hierarchical data representation and novel consistency metrics are generalizable and compatible with technologies such as prompting based data
generation using LLMs which opens the opportunities for future research on scaling up the dataset. We collect picture story data from two resources emphasizing on Indian and African backgrounds respectively. With such inputs, we encourage crossculture understanding. The utilization of stories designed specifically for young children further injects cultural neutrality, as the contents of the stories focus on core concepts appealing to children universally and have low cultural complexity.

heading: 7 Conclusions
text: We present a novel dataset for systematic assessment of multi-modal comprehension engines. Inspired by Bloom’s Taxonomy from education research, we collect multiple-choice samples based on picture story data from creative commons resources. Each sample is labeled with a cognitive skill required in solving the multiple-choice task and is associated with a specific level of comprehension from Bloom’s Taxonomy. We further propose a novel hierarchical graph representation describing knowledge extracted from picture stories at different Bloom’s levels. The proposed Story Graph naturally extends the concept of scene graphs and maps data into the hierarchical taxonomy. We demonstrate that automatic data augmentation can be achieved by traversing through the underlying graph. With the proposed dataset, we can not only assess given models in a graded manner but also characterize the models with respect to consistency of their performance. The proposed data structure and metrics pave the way for a wide range of interesting future works, where hierarchical graph representation can be used to guide systematic data storage, retrieval and generation.

heading: Acknowledgements
text: This work was supported in part by NSF awards #1909696 and #2326491. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies or endorsements of any sponsor.

heading: A Appendix
text: A.1 Data Collection We provide a web-based UI for data collection as shown in Figure. A1. We recruit annotators in the United States via Amazon Mechanical Turk. While our tasks are complicated, we empirically adjust our instructions, UI design and collection protocols with iterations of data collection where we increase the payment based on the quality of the inputs. We report in Table A1 the examples of templates constructed based on corresponding cognitive skills from Bloom’s Taxonomy for the data collection.
A.2 Example data In Figure A.2, we show a complete picture story as one example from BloomVQA. In Table A2, we show a corresponding set of example questions and answers from different Bloom’s levels collected based on the story.
L ev
el Sk
ill Te
m pl
at e
1 re
m em
be r
W ha
ti s/
ar e
<c ha
ra ct
er ><
ac tio
n> <l
oc at
io n> ? 2 cl ar if y H ow w ou ld yo u cl ar if y th e m ea ni ng of <c ha ra ct er >< ac tio
n> <l
oc at
io n> ? 3 ap pl y W ha tw ou ld yo u do to ch an ge th e <a ttr ib ut e> of <c ha ra ct er >? 4 co m pa re H ow w ou ld yo u co m pa re th e <a ttr ib ut e> of <c ha ra ct er > be fo re an d af te r< ch ar ac te r> <a ct
io n>
<l oc
at io
n> ?
5 cr
iti qu
e H
ow w
ou ld
yo u
cr iti
qu e
th e
st ra
te gy
<c ha
ra ct
er >
us ed
to <a
ct io
n> ?
6 im
pr ov
e H
ow w
ou ld
yo u
im pr
ov e
th e
st ra
te gy
<c ha
ra ct
er >
us ed
to <a
ct io
n> ?
Ta bl
e A
1: E
xa m
pl es
of te
m pl
at es
an d
co rr
es po
nd in
g co
gn iti
ve sk
ill s
at di
ff er
en tB
lo om
’s le
ve ls
.
Figure A1: We designed a Web-based UI with general instructions and detailed instructions at each Bloom’s level provided to annotators without background expertise in the domain.
Figure A2: Example picture story: "Foxy joxy plays a trick"
L ev
el Sk
ill Q
ue st
io n
A ns
w er
s
1 re
m em
be r
W ha
ti s/
ar e
Fo xy
Jo xy
se lli
ng in
fo re
st ?
fa ke
w at
er m
el on
s (C
or re
ct )
fr ee
w at
er m
el on
s (I
nc or
re ct ) re al w at er m el on s (I nc or re ct ) fr es h fr ui ts (I nc or re ct )
2 cl
ar if
y C
an yo
u de
sc ri
be w
ha th
ap pe
ne d
w he
n Fo
xy Jo
xy se
lli ng
w at
er m
el on
s in
th e
fo re
st ?
he so
ld at
hi gh
pr ic
e (C
or re
ct )
he us
ed no
tr ic
ks (I
nc or
re ct ) he w as ho ne st (I nc or re ct ) he m ad e th e el ep ha nt ha pp y (I nc
or re
ct )
3 ap
pl y
W ha
tw ou
ld yo
u ch
oo se
if Jo
xy tr
ie d
to se
ll yo
u w
at er
m el
on s
at su
rp ri
si ng
ly lo
w pr
ic e
in th
e fo
re st
?
do ub
ta bo
ut th
e pr
ic e
(C or
re ct ) bu y it w ith ou tq ue st io ns (I nc or re
ct )
bu y
it an
d th
en se
ll it
to ot
he rs
(I nc
or re
ct )
be ha
ve lik
e ot
he ra
ni m
al s
(I nc
or re
ct )
4 an
al yz
e W
ha ti
s th
e fu
nc tio
n of
an im
al ga
th er
in g?
co m
in g
up a
pl an
to pu
ni sh
Jo xy
(C or
re ct ) ga th er in g w ith ou tt he ow l( In co rr ec t)
di sc
us s
ho w
to ca
re ab
ou tJ
ox y
(I nc
or re
ct )
ta ke
a le
ss on
ab ou
te co
no m
ic (I
nc or
re ct
)
5 ev
al ua
te E
va lu
at e
w he
th er
un io
n of
an im
al is
su cc
es sf
ul or
no tt
o ac
hi ev
e st
re ng
th of
un ity
.
ye s
be ca
us e
th ey
ta ug
ht Jo
xy a
le ss
on (C
or re
ct )
N o
be ca
us e
Jo xy
ne ve
rc he
at ed
(I nc
or re
ct )
N o
be ca
us e
th ey
go tfl
uf fy
w at
er m
el on
(I nc
or re
ct )
Y es
be ca
us e
th e
an im
al s
w er
e w
el lp
ai d
(I nc
or re
ct )
6 im
pr ov
e H
ow w
ou ld
yo u
de ve
lo p
a di
ff er
en tw
ay to
so lv
e an
im al
’s pr
ob le
m :g
ot ch
ea te
d by
Jo xy
.
se nd
hi m
to po
lic e
(C or
re ct ) bu y m or e w at er m el on s (I nc or
re ct ) bu y ot he rf ru its (I nc or re ct )
ne ve
re at
w at
er m
el on
ag ai
n (I
nc or
re ct
)
Ta bl
e A
2: E
xa m
pl e
qu es
tio ns
an d
an sw
er s
ov er
di ff
er en
tB lo
om ’s
le ve
ls on
th e
pi ct
ur e
st or
y "F
ox y
jo xy
pl ay
s a
tr ic
k"

heading: Future Work
text: The proposed data structure and metrics pave the way for a wide range of interesting future works, where hierarchical graph representation can be used to guide systematic data storage, retrieval and generation