The Generative AI Paradox: “What It Can Create, It May Not Understand”
Authors: Peter West
Ximing Lu
Nouha Dziri
Faeze Brahman
Linjie Li
Jena D. Hwang
Liwei Jiang
Jillian Fisher
Abhilasha Ravichander
Khyathi Chandu
Benjamin Newman
Pang Wei Koh
Allyson Ettinger
Yejin Choi
Venue: ICLR 2024 poster (2024)
Keywords: LMs
language models
vision models
GPT4
ChatGPT
Midjourney
generative models
understanding models
Primary area: generative models
TL;DR: We propose and test the Generative AI Paradox, which posits that generative models, unlike humans, acquire generative capabilities that are not contingent upon their ability to understand those same types of outputs

Abstract: The recent wave of generative AI has sparked unprecedented global attention, with both excitement and concern over potentially superhuman levels of artificial intelligence: models now take only seconds to produce outputs that would challenge or exceed the capabilities even of expert humans. At the same time, models still show basic errors in understanding that would not be expected even in non-expert humans. This presents us with an apparent paradox: how do we reconcile seemingly superhuman capabilities with the persistence of errors that few humans would make? In this work, we posit that this tension reflects a divergence in the configuration of intelligence in today's generative models relative to intelligence in humans. Specifically, we propose and test the **Generative AI Paradox** hypothesis: generative models, having been trained directly to reproduce expert-like outputs, acquire generative capabilities that are not contingent upon---and can therefore exceed---their ability to understand those same types of outputs. This contrasts with humans, for whom basic understanding almost always precedes the ability to
generate expert-level outputs. We test this hypothesis through controlled experiments analyzing generation vs.~understanding in generative models, across both language and image modalities. Our results show that although models can outperform humans in generation, they consistently fall short of human capabilities in measures of understanding, as well as weaker correlation between generation and understanding performance, and more brittleness to adversarial inputs. Our findings support the hypothesis that models' generative capability may not be contingent upon understanding capability, and call for caution in interpreting artificial intelligence by analogy to human intelligence.

Published as a conference paper at ICLR 2024
THE GENERATIVE AI PARADOX:
"What It Can Create, It May Not Understand"
Peter West Ximing Lu Nouha Dziri
Faeze Brahman 1.2+
Linjie Li
Jena D. Hwang²
Liwei Jiang'? Jillian Fisher'
Abhilasha Ravichander
Khyathi Raghavi Chandu
Benjamin Newman'
Pang Wei Koh Allyson Ettinger* Yejin Choil.2
Universityy of Washington Allen Institute for Artificial Intelligence (pawest, linjlijes washington edu (ximinglu, nouhad, faw.eb)allenai.oomm
ABSTRACT
The recent wave of generative AI has sparked unprecedented global attention, with both excitement and concern over potentially superhuman levels of artifi- cial intelligence: models take only seconds produce outputs that would challenge or exceed the capabilities even of expert humans. At the same time, models still show basic errors in understanding that would not be expected even in non-expert humans. This presents us with an apparent paradox: how do we rec- oncile seemingly superhuman capabilities with the persistence of errors that few humans would make? In this work, we posit that this tension reflects a divergence in the configuration of intelligence in today's ss generative models relative to intel- ligence in humans. Specifically, we propose and test the Generative AI Paradox hypothesis: generative models, having been trained directly to reproduce expert- like outputs, acquire generative capabilities that are not contingent upon-and and can therefore exceed-tiie ability to understand those same types outputs. This contrasts with humans, for whom basic understanding almost always precedes the ability to generate expert-level outputs. We test this hypothesis through controlled experiments analyzing generation understanding in generative models, across both language and image modalities. Our results show that although models can outperform humans in generation, they consistently fall short of human capabili- ties in measures of understanding, showing weaker correlation between generation and understanding performance, and more brittleness to adversarial inputs. Our findings support the hypothesis that models' generative capability may not be con- tingent upon understanding capability, and call for caution in interpreting artificial intelligence by analogy to human intelligence.
INTRODUCTION
"What cannot create, I do not understand." - Richard Feynman
The recent wave of generative AI, from ChatGPT to GPT4 DALL-E 2/3 Midjourney, has sparked unprecedented global attention-with equal parts excitement about the expansive potential applications, and deep concern about the dangers of "iintelligence"" that seems even exceed that of humans. Indeed, in both language and visual domains, current generative models take only seconds to produce outputs that could challenge experts with years skill and knowledge, providing com- pelling motivation for claims that models have surpassed human intelligence (Bubeck et al. 2023 Surameery Shakor 2023). At the same time, probing of models' outputs continues to uncover ba- sic errors in understanding that would be unexpected even for non-expert humans (Dziri al.) 2023 Arkoudas 2023 Qin al. 2023). This presents us with an apparent paradox: how do we reconcile
First co-authors.
"Intelligence" and "understanding" here refer particularly demonstrable aspects of models and technol- ogy (as in "Artificial Intelligence" or "Natural Language Understanding").

---

Published as a conference paper at ICLR 2024
A.
Language Modality
B.
Vision Modality
D.
Austin had a rough day at work and decided go the bar.
Write paragraph story aboutt 3 people ouilding a houss
blue backpack and red orange
mountain gooat stands on top of rock on hill
Austin drink that night. Why dld Austin de this?
Generation
Austin likely went the bar and had chink becsuse his had rough uneind,
[..]] Finalyy, after years of seving money
designs, they theam house. Bebs with his architectural expertise,
wanted
dicidded
or temporantyy escape from the atreas
and frustrations he eogae seeceedd during day.
sketched broothtaking design storied country house.
Under Understanding
How many total designs were sketched the story?
Which image matches the caption?
choices provided:
A. Settle down 12
Is mountain gowt?
B. Go to the bar Order
agedeflateeer Many
Espaned Ves
story only montors one desigg baiing slorcheel [...]
Dasedion image
to
[snlectecc]
that 1t not mountalie goad. ior.]
(Superi SEmMIG
Ohiwezsione Sermioo
[Seaacin Sa 188))
DesseCRsATE Sa Sarreoo
Figure 1:
Generative Al in language and vision can produce high-quality generations. Paradoxically, however, models have trouble demonstrating selective (A,C) or interrogative (B,D) understanding of these modalities.
the seemingly superhuman capabilities of these models with the persistent presence of fundamental errors that most humans could correct?
We posit that this tension arises because the configuration of capabilities in today's ' generative mod- els diverges from the configuration of intelligence in humans. Specifically, in this work we propose and test the Generative AT Paradox hypothesis: generative models, having been trained directly to reproduce expert-like outputs, acquire generative capabilities that are not contingent upon-and and can therefore their ability to understand those same types of outputs. This contrasts with hu- mans, for whom basic understanding nearly always serves as a prerequisite to the ability to generate expert-level outputs (Gobet 2017 Alexander 2003 Berliner 1994).
We test this hypothesis through controlled experiments analyzing generation and understanding ca- pabilities in generative models, across language and visual modalities. We conceptualize "under- standing" relative to generation via two angles: 1) given a generative task, to what extent can models select correct responses in discriminative version of that same task? and 2) given correct gener- ated response, to what extent can models answer questions about the content and appropriateness of that response? This results in two experimental settings, selective and interrogative, respectively,
Though our results show variation across tasks and modalities, a number of clear trends emerge. In selective evaluation, models often match or even outperform humans on generative task settings, but they fall short of human performance in discriminative (understanding) settings. Further analysis shows that discrimination performance more tightly linked to generation performance in humans than in GPT4, and human discrimination performance is also more robust to adversarial inputs, with the model-human discrimination gap increasing with task difficulty. Similarly, in interrogative eval- uation, though models can generate high-quality outputs across tasks, observe frequent errors in models' ability to answer questions about those same generations, with model understanding perfor- mance again underperforming human understanding. We discuss number of potential reasons for this divergence in capability configurations for generative models versus humans, including model training objectives, and size and nature of input.
Our findings have a number broader implications. First, the implication that existing conceptual- izations of intelligence, derived from experience with humans, may not be able to be extrapolated to artificial inteligence--llhough Al capabilities in many ways appear to mimic or exceed human intelligence, the contours of the capability landscape may diverge fundamentally from expected pat- terns in human cogmition. On the flip side, our findings advise caution when studying generative models for insights into human intelligence and cognition, as seemingly expert human- like outputs may belie non non-human-lia mechanisms. Overall, the generative Al paradox encourages studying models as an intriguing counterpoint human intelligence, rather than as a parallel.
THE GENERATIVE AI PARADOX
We begin by outlining the Generative AI Paradox and experimental design to test it.
2

---

Published as a conference paper at ICLR 2024
2.1 OPERATIONAL DEFINITIONS
Figure [ offers examples of the seemingly paradoxical behavior of generative models. In language (column B), GPT4 is able generate a compelling story about friends building a house, but when pressed on details of its own generaled story, fails to correctly answer a simple question: GPT4 asserts that only one design was sketched in the story despite writing about years of "sketching designs". In vision (column C), a generator produces a correct image beyond average human capa- bilities, yet the understanding model is unable single out that correct generation against plausible alternatives, despite selection being the seemingly "easier" task. In both cases, models meet or exceed human generation abilities but lag in understanding.
Observations such as these motivate the Generative Al Paradox:
Generative models seem to acquire generation abilities more effectively than un- derstanding, in contrast to human intelligence where generation is usually harder.
Testing this hypothesis requires an operational definition of each aspect of the paradox. First, what it means for generation to be "more effective" than understanding for given generative model mg- understanding model m, and task t, with human intelligence as a baseline. Taking and u to be some performance measures of generation and understanding, formally state the Generative Al Paradox hypothesis as:
g(human, t) = g(m,, t) = u(human, t) um,,,t)
(1)
Put simply, the hypothesis holds for a task if human who achieves the same generation perfor- mance g as a model m, would be expected to achieve significantly (> e for a reasonably large e) higher understanding performance than a model m, does] In simpler terms, models perform worse understanding than we would expect of humans with similarly strong generative capabili- ties. In the language domain,
Generation is straightforward to operationally define: given task input (question/prompt), gen- cration the production of observable content to satisfy that input. Thus, performance g can be evaluated automatically or by humans (e.g. style, correctness, preference). While understanding is not defined by some observable output, it can be tested by explicitly defining its effects. Thus, we measure performance by asking the following questions:
1. Selective evaluation. For a given task, which can be responded to generatively, to what extent can models also select accurate answers among provided candidate in discriminative version of that same task? A common example of this is multiple choice question answering, which is one of the understanding in language models (Wang et al. 2219)) (Figure [ columns A, C). This tests the performance aspect of understanding, i.c. the ability identify the answer
common ways examine both human understanding and natural language
human input.
2. Interrogative evaluation. For given generated model output, to what extent can models accu- rately respond to questions about the content and appropriateness of that output? This akin to an oral examination in education (Sabin al., 2021). (Figure columns B, D This tests the explainability aspect of understanding. i.e. the ability to comprehend one's s own answer.
These definitions of understanding provide us with blueprint for evaluating the Generative AI Paradox, allowing us to test whether Hypothesis holds across modalities, tasks, and models.
2.2
EXPERIMENTAL OVERVIEW
Here, we provide high-level road map for experiments informed by the definitions above. We propose 2 sub-hypotheses to test across experimental settings, and provide cross-experiment details.
2.2.1
HYPOTHESES
Evaluating whether Hypothesis [ holds for given task requires establishing a human baseline, specifically, the understanding performance expect from human with the same generation
TTo clarify, the paradox hypothesis is not restricted to the use of single model assess both generative and understanding capabilitites; different models can he employed test these aspects independently.
3

---

Published as a conference paper at ICLR 2024
pabilities the model. We define how such baseline is established for both kinds of understanding above, resulting in 2 sub-hypotheses.
Selective evaluation. Here, we explicitly measure human generation and understanding perfor- mance to establish a baseline. We say Hypothesis Tholds if models underperform in understanding compared to humans with equivalent generation performance (or lower generation performance, as- suming that if humans matched model generation they would do even better at understanding. The sub-hypothesis simply:
sub-hypothesis 1: models meet exceed humans at generation while lagging at discrimination.
Interrogative evaluation. For the human baseline here, we assume that humans can answer sim- ple questions of understanding about their own generations. For a given task input, we test how accurate models are at answering questions on Al generated outputs and as the human baseline, as- sume near pear-perfect accuracy on such questions for their own generations. The sub-hypothest in this case is:
hub-hyyothesis 2: models struggle to answer simple questions about generated content, which hu- mans could answer for their own generations.
2.2.2 MODELS AND EXPERIMENTS
We focus our study on the strongest current generative models, i.e., those driving interest and con- cern among experts and the public. We investigate language and vision, modalities where recent impressive progress has been made. We test language models for both generative and understanding capabilities given strong performance in both areas, i.e. taking m, = mg- We test GPT4 (gpt-4) and GPT3.5 (GPT3. 5-turbo) in zero-shot setting where we instruct models to output response given some background information ($A). In contrast, for vision, image generators show weaker un- derstanding (Li et al. 2023a) than dedicated understanding models, and so assume m, my for vision. We use Midjourney (Inc. 2023) to generate, CLIP (Radford et al.) (02T)) and OpenCLIP (II-) harco al. 2022]) as understanding models for selective evaluation, and BLIP-2 (Li al. 2023b), BingChat (Microsoff) 2023), and Bard (Google, 2023) for interrogative evaluation. All results on
vision models are obtained in zero-shot fashion.
We conduct experiments across both sub-hypotheses, investigating tasks with selective evaluation of understanding (sub-hypothesis in $3 and investigating tasks with interrogative evaluation of understanding (sub-hypothesis 2) in $4 Both sections include both language and vision tasks.
3 CAN MODELS DISCRIMINATE WHEN THEY CAN GENERATE?
First, in selective evaluation, conduct side-by-side performance analysis generative and discriminative variants of tasks to assess models' generation and understanding capabilities in language and vision modalities. We compare this generative and discriminative performance that of humans. For our tasks we draw on diverse source benchmarks, detailed below:
Language benchmarks. For dialogue, we explore two open-ended PitasetssMuutall Cui ct al. 2020) and DREAM (Sun et al. 2019). and a document-groundee benchmark, Faithdial DDziri al. 2022). These tasks require generating coherent continuations based conversation history (faithful to the document in grounded dialogue). For reading comprehension, we include Topioca (Adlakha al. 22022; conversational QA) and RACE (Lai et al. 2017 factual QA). For summariza- lion, we consider XSUM (Narayan et al.] 2018). We also include the commonsense benchmarks CommonSenseQA (Talmor et al. 2019), SocialIQA (Sap al. 2019). HellaSwag (Zellers et al., 2019), PIQA (Seo et al. 2018), and aNLG/oNLI (Bhagavatula al. (020). Lastly, consider the natural language inference tasks WaNLI (Liu et al. 2022) and 8-NLI (Rudinger et al. 2020).
Vision benchmarks. For image generation, we source text prompts from four benchmarks: these range from descriptions of natural scenes, (likely in-domain for the model) to out-of distri--ittiitttionn scenes with specific attributes and relationships that rarely exist in real images. Prompts sourced from: COCO (Lin et al. 2014). PaintSkill (Cho et al. 2022). DrawBench (Saharia et al., 2022)
and TeeCohpBench (Huang al. 2023). More dataset details are in $A.2
4

---

Published as a conference paper at ICLR 2024
Commonsense
1,00
877
.97
.92
.89
.90
.86.85
87
86
.22
.97
82
.92
.97
.22
0.75
.72
.70
70
0.50
CSQA
SocialiQA
HellaSwag
PIDA
D-NLVG-NLG
Dialogue
1.00
92
.92
187
.83
86
96
96
.97
90
.66
.73
62
.77
63
.43
WaNLI
6-NLI
FalthDial
DREAM
Mutual+
Summariizatin
Conv,
QA
1.0
vs. Human
1.00
.99
84
.85
92
82
.94
.6
89
.4
.00
.66
Generation
0.75
.66
:64
0.5
0.50
XSUM
Topioca
RACE
0.0
Generative
Discriminative
0.0
0.5
1.0
GPT4
GPT3.5
Discrimination
Figure 2: Discriminative and generative performance of GPT3.5 and GPT4 vs Humans. Models outperform
humans in generation but underperform them in discrimination for most of the cases. The scatter plot in the bottom right summarizes GPT4's performance human performance (using the hard negatives from Section 3.2 to measure discriminative accuracy XSUM and FaithDial); each point represents diffcrent task. Hu- mans have a larger positive slope between discrimination and generation abilities compared GPT4.
1.0
.33
.99
1.0
.97
0.8
82
.80
.84
.84
.79
.84
.79
Midjourney (generative)
CLIP (discriminative)
.63
OpenCLIP (discriminstive)
0.6
Human (discriminative)
COCO
PaintSkil
DrawBench
T21-CompBench
Figure 3: Model and human performance under the generative and discriminative settings on the vision modal- ity. We observe models fall short of human accuracy in discriminative performance, and their generative accu- racy also lags behind their diseriminative accuracy.
Experimental setup. For each task and modality, we consider two settings: i) generative: we prompt models to generate a response given task-specific inputs (e.g., dialogue history, document, image caption), and ii) discriminative: we require task-specific models to select the correct answer from of candidates, using existing candidates where available and otherwise generating options.
For the generative setting, we conduct human evaluations using Amazon Mechanical Turk (AMT) judge the correctness of the generated responses (i.e, text or image) and report percentage of successful responses satisfying task requirements. For example, for the language domain, we present humans with examples from the language benchmarks.
For the discriminative setting, we report the accuracy of choosing the ground-truti response among the candidate options. To establish human performance baseline, we ask workers perform all discriminative tasks and evaluate the correctness of the ground-truth responses for each task Details of AMT annotations and instructions are in $D
3.1 GENERATIVE AND DISCRIMINATIVE CAPABILITIES IN MODELS VS. HUMANS
Language. Figure presents comparison of GPT3.5, GPT4 and human generative and discrim- inative performances. We see that for 10 the 13 datasets, Sub-hypothesis supported at
"Ground-truhh responses were initially written by humans for the language tasks, while ground-truth images generated by Midjourney.

---

Published as a conference paper at ICLR 2024
Language
Vision
1.0
.98
.84
.63
.69
GPT4 (discriminative)
0.5
OpenCLIP (discriminative)
Human (discriminative)
Hard
Hard
Hard
Hard
FaithDial
DrawBerch
T2l-CompBBnnch
Figure 4: Model vs. human performance across varying levels of answer difficulty on discriminative tasks.
least one model, with models outperforming humans in generation but underperforming humans in discrimination. For 7 of the 13 datasets, this sub-hypothesi is supported in both models.
Vision. It is not practical to ask humans produce detailed images as we do with vision models, but we assume that an average human could not achieve the stylistic quality of models like Midjour- ney and thus assume human generation performance is lower. Therefore, we only compare models' generative and discriminative accuracy humans' discriminative accuracy. Similar to the language domain, Figure 3 shows that CLIP and OpenCLIF[ fall short of human accuracy in discriminative performance. Assuming human generation is worse, this agrees with sub-hypothesis 1: Vision AI exceeds average humans at generation but lags at understanding.
3.2 MODELS FALL FURTHER SHORT OF HUMAN PERFORMANCE WITH HARDER DISCRIMINATION TASKS
We take a closer look at the gap in discriminative performance between humans and models by manipulating the difficulty of the negative candidates. Two types of negatives are considered: Hard negatives: challenging examples that deter models from relying on data biases and artifacts to produce an answer. These negatives are wrong subtle and challenging ways; recognizing them may require profound understanding of the task. ii) Easy negatives: these candidates are semantically distant from the topic the question, providing a clear contrast to the correct answer.
Figure (left) shows the comparison between GPT4 and humans Notably, as the complexity of the candidate answers increases, model performance gradually declines. For instance, in the XSUM task, GPT4 achieves 100% accuracy when selecting the correct answer from easy negatives, but this drops to 19% when confronted with hard negatives. XSUM exhibits substantial difference in performance compared to FaithDial. Upon inspection, we observe that models tend to make the most mistakes in discrimination tasks when the responses lengthy and challenging, such as summarizing lengthy documents. In contrast, humans can maintain a consistently high level of accuracy across different levels of difficulty.
Figure (right) shows the discriminative perfor- mance of OpenCLIP, in comparison to humans, across difficulty levels. Consistent with the language results, and even more robustly across tasks, we see that while humans show versatile performance across hard and easy negative settings, model per- formance drops substantially when confronted with hard negatives (from 100% to ~69%). Overall, these results highlight that humans have the ability to dis- cern correct answers even when faced with challeng- that this
GPT4
Human
Both
0.8
0.6
0.4
0.2
0.0
CSoA
SecialigA
Hellswwnn
WNLG
fatthDal
DRE:
M
WoNL
B-HLJ
XSUM
Tepiace
RACE
ing adversarial examples, but pability not as robust in LMs. This discrepancy raises questions about the true extent of these mod- els' understanding.
Figure 5: Human's preference scores between
human-generated vs. GPT4-generated responses
"We report the best results on CLIP (c1i ip-vi terittllaae www.itttrrgepattteee 14) and OpenCIIP (CLIP-ViT-big) 11a-1a on2 31--399-11166 60k), more results can be found in {B.3
*See {B.2 for details about negative candidates construction. For language domain, hard negatives are constructed only for tasks that are originally generative in nature (i.c., FaithDial and XSUM). 'The same trend also applies for GPT3.5.

---

Published as a conference paper at ICLR 2024
3.3 MODEL GENERATIONS ARE PREFERRED OVER HUMAN GENERATIONS
To better understand the gap between humans and language models, we asked AMT workers to provide their preferences between machine and tasks, along with a rationale for their choice While both sets of responses score high in correctness (Figure 2) Figure [S] shows notable trend: workers often favor responses from GPT4 over those generated by humans. The same applies for GPT3.5 (Figure[[]) Ii SB.3). The rationales provided by humans often indicate a preference for GPT4 due to longer response length, more elegant writing style, and being more informative, while human choice is preferred for brevity and conciseness (Figure 12 in $ This makes the divergence in capabilities models excelling in relative terms at generation and humans at understanding--uut tasks-even more apparent.
answers in the language-related
CAN MODELS UNDERSTAND WHAT MODELS GENERATE?
In the previous section, showed that models often excel at generating accurate answers while lagging behind humans in the discriminative task. Now, in our interrogative evaluation, we investi- gate to what extent models can demonstrate meaningful understanding of senerations-somelee humans are highly capable of-by directly asking models questions about generated content.
Language experimental setup. In language, we first prompt models generate a paragraph ing task-specific background information. Then using its generation as context, we ask the model multiple-choice questions about its own generated information [ For example, for XSUM (Narayan et al. 2018) (summarization) we prompt the model to generate an article based on a ground-truth summary, and then ask the model select the best summary (same choices as $3) for the generated article. For Mutual* (Cui et al. 2020) (dialogue), the model generates the conversation history that leads to a given dialogue, and then is asked to choose the best dialogue continuing that history. In HellaSwag (Zellers et al. 2019) (commonsense), the model generates the context preceding a given sentence and then selects the most fitting continuation for that generated context. We only perform selective evaluation on the correct generations verified by humans.
We use zero-shot GPT3.5 and GPT4 for all of the evaluations, both generating and question an- swering. We report the model generation performance, the selection performance based content generated by the model, and human selection performance using the model's generated content. As an implicit baseline, we assume that humans can answer such questions about their own generations with high accuracy, and so refrain from the complex process of eliciting these human generations.
Vision experimental setup. We conduct interrogative evaluation on image understanding models via visual question answering in an open-ended setting. We consider TIFAy1.0 (Hu et al. 2023) as the evaluation benchmark, with text prompts from COCO, PaintSkill, Draw Bench and Parti (Yu) et al. 2022). TIFAv1.0 1.0 includes questions automatically generated by a language model, only con- cerning the content specified in the text prompt (e.g., about existence/atrriutes of an object and relative position between objects). We first ask Midjoumey to generate images, based on the text prompts. Then, we interrogate the understanding models (e.g., BLIP-2) with answerable questions (verified by AMT workers) about the generated images. AMT is used to collect human responses, and judge the correctness of human/model outputs. See $C. for more details.
Results. Results for the language modality are shown in Figure O (left). We observe that while the models excel at generation, they make frequent errors in answering questions about their own generations, indicating failures in understanding. Humans, who we assume could not generate such text at the same speed or scale, consistently achieve higher accuracy in QA compared to the model, despite the fact that questions are about the model's own output. As stated in sub-hypothesis 2, we expect humans would achieve even higher accuracy for their own generations. We note that the humans in this study not experts; producing text as sophisticated as the model's output could be significant challenge. We anticipate that the performance gap in understanding one's own generation would widen even more when comparing the model human experts, who are likely to answer such questions with near-perfect accuracy.
"See Figure 12 in $ B.3 for details.
Unlike questions here are ahout the generation, rather than taking the generation as potential answer.
7

---

Published as a conference paper at ICLR 2024
Language
Vision
1,0
99
.93
.95
.84
.65
.84
.87
HellaSwag
COCO
Generative
GPT4
Midjourney
QA
GPT4
Bard
BingChat
BLIP-2
Human
Figure 6: Models vs, human performance on language/visual QA based model generated texts/images.
Figure 6 (right) shows the interrogative results in the visual modality We see that image under- standing models still fall short of human accuracy in answering simple questions about elements in the generated images. At the same time, state-of-the-ard image generation models can generate images quality and speed beyond most average humans (who we expect will have trouble gen- erating comparable realistic images), indicating a relative gap between generation (stronger) and understanding (weaker) in vision Al compared to humans. Surprisingly, the performance gap be- tween models and humans is smaller for simpler models than advanced multimodal LLMs (i.e., Bard and BingChat), which have some intriguing visual understanding abilities, but still struggle to answer simple questions about generated images.
DISCUSSION
Assessing the generative AI paradox. Broadly, we find significant experimental evidence of the Generative AI Paradox: though models regularly outperform humans in text and image genera- tion, they fall short of human performance in discriminative versions of generative tasks, and when answering questions about generated content. Furthermore, our analyses show that discrimination performance is more tightly linked to generation performance in humans than in GPT4, and that human discrimination performance is also more robust challenging inputs. These trends vary across tasks and modalities, but in general our results robustly support the hypothesis that generative capability can outstrip understanding capability in models, especially compared with humans.
Proposed explanations and points of future study. Given the above evidence in support of the Generative AI Paradox, the next question is: what factors could lead to models that excel gen- eration even when they cannot demonstrate strong understanding? We propose some hypotheses below, and encourage future work to explore this question.
Generative AI is defined by the generative learning objective, explicitly encouraging reconstruc- tion/generation the training distribution, while only implicitly encouraging understanding furthers this goal. Human learning, while not completely understood. likely diverges from this by encouraging behavior beyond pure reconstruction stimuli.
Although often query generative models if they medium (e.g. text over many authors in language models). Providing context may push models closer to emulating specific individual (Andreas) (022), but they tend towards behavior that looks distributionally correct rather than individually correct, prioritizing stylistic and document-wide features details necessary for understanding tasks. Training many documents (e.g. huge swaths of internet text) also contrasts with humans: it would take an average human reader e.g. over 32 years just read all the pages of Wikipedia (contributors) Brysbaent] 2019). This obvious discrepancy in not only quantity, but also diversity of knowledge could encourage models to use existing solutions to problems, which they have seen already, whereas humans have not and therefore need to exercise understanding and reasoning to answer the same questions correctly.
individuals, they typically model
Evolutionary and economic pressures can affect the way that Al develops. For instance, popular language model architectures have shown preference for languages like English (Ravfogel et al.) 2019) which has seen the most attention NL. (Bender 2019) and thus the most reward for im-
We report performance of subsets, results can he found in 2
BingChat, Bard and the best BLIP-2 model (a----a--ntt5
two
8

---

Published as a conference paper at ICLR 2024
provement. Similar pressures could encourage architectures, training paradigms, and other deci- sions that favor generation over understanding, as generation is harder for humans and thus more useful/valuable. Designing systems that are not affected by the Generative Al Paradox will require understanding its cause. Given the potential explanations above, promising paths forward may in- volve alternative optimization objectives, limiting the memorization in models to force reasoning. and even incentivizing stronger understanding at field level.
Dataset/benchmark contamination a potential limitation with proprictary models, but this should have similar effects on generation and discriminative evaluation in & and our eval- uation in $4 uses novel generations which would not be seen training time. Also, we focus on a small set of the most hapular/widelyy used models. Future work should investigate a wider range of models, including smaller or weaker models, for which we hypothesize the paradox may be even more pronounced as we often saw with GPT3.5 vs GPT4 ($3)
Limitations.
While our evaluation of human performance focused, future work can explore more extensive comparisons between model and human performance. We also advocate for adopting comparison to humans as a widespread practice, to carefully judge when model capabilities extrapolate with human capabilities, and when they do not. Finally, we only investigate one divergence between humans and models. Proposing and testing other points of divergence between artificial and natural intelligence exceeds our scope but will be imperative to calm concerns and calibrate excitement.
6 RELATED WORK
Generative paradoxes in large language model behavior. Prior work paradoxically employs large language models to improve their own generations, finding that models successfully identify mistakes (despite these mistakes being generated by the models themselven, MMddan et al. (2023) prompt models to critique and improve their own generations. [Agraaal et al. (2023) find that models can identify hallucinated content in their own generations, Gero et al. (2023) show that models can identify erroneously omitted elements in generated in clinical extraction data.
Inconsistencies in large language models. Past work suggests that large language models (LMs) lack robust concept representation. Dziri et al. (2023) show that strong models often struggle at solving basic tasks like multiplication. Elazar al. (2021) and Ravichander al. (2020) show that LMs make inconsistent predictions when prompted with similar statements. Ribeiro et al. (2019)
find that QA systems often generate contradictory answers. Kassner & /Schützee[(00)) and Ettinger (2020) find that models can generate correct facts but also their negations. (Jang et al. (2022) con- struct benchmark showing large LMs often make inconsistent predictions. Berglund et al. (2023) demonstrate that while models can correctly recognize factual knowledge present in their training data, they fail to make inferences related to those facts.
Generative models and human cognitive mechanisms. While the reasoning mechanism of mod- els is unknown, prior work has investigated models possess similar competencies with humans. Stojnić et al. (2023) evaluate commonsense psychology. finding that while infants can reason about the causes actions by an agent, models are not capable cannot emulating this. [Sap et al. (2022) find that language models fail to demonstrate Theory-of-Mind [Storks et al. (2021) and Bisk et al. (2020) show discrepancies between human and model capacities in physical commonsense reasoning.
7 CONCLUSIONS
In this work, we propose the Generative AI Paradox hypothesis, which posits that impressive gener- ation abilities in generative models, by contrast to humans, may not be contingent upon commensu- rate understanding capabilities. We test this through controlled experiments language and vision modalities, and though our results show variation depending on task and modality, we find robust support for this hypothesis. Our findings have a number of broader implications. In particular, they imply that existing conceptualizations of intelligence, as derived from experience with humans, may not be applicable to artificial intelligence although AI capabilities may resemble human in- telligence, the capability landscape may diverge in fundamental ways from expected patterns based on humans. Overall, the generative Al paradox suggests that the study of models may serve as an intriguing counterpoint to human intelligence, rather than a parallel.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was funded in part by NSF (DMS-2134012), DARPA MCS program through NIWC Pacific (N66001-19-2-4033 Darpa SemaFor, and the Allen Institute for Al. We thank OpenAl for offering access to various models through the API.
REPRODUCIBILITY
We include simple description of overall details in $2 well experiment-specific details like datasets used and evaluation setup at the beginning of each experiment section, $ and $ These descriptions are relatively brief, and include more extensive information in the appendix. For instance, we include more detail on models, model settings, and datasets in { We also include more experimental details and further experiments that can be useful for work comparing and reproducing our results in $ and $ Finally, we include more extensive information about our human evaluation templates in DD All datasets and models we use here are public or can be accessed through public interfaces.
ETHICS STATEMENT
Our work is conducted using existing benchmarks and models, and does not introduce new data, methodology, models with significant risk of harm. All experiments we conduct would be con- sidered analysis of existing resources, particularly in terms of the performance of models. We con- duct human studies, with appropriate IRB exemptions. Based on estimates of the time for task completion, we ensure workers are paid at least $15 USD per hour. We strive to not conduct any experiments that introduce additional bias, harm, reduction in diversity, either through the way our research is conducted or its effects. We acknowledge that our work is primarily concerned with certain aspects of performance and does not specifically measure concepts such bias toxicity.
REFERENCES
Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva Reddy. Topi- ocqa: Open-domain conversational question answering with topic switching. Transactions of the Association for Computational Linguistics, 10:468 483,, 2022.
Ayush Agrawal, Lester Mackey, and Adam Tauman Kalai. Do language models know when they're hallucinating references? arXiv preprint arXiv:2305. 18248, 2023.
Patricia A Alexander. The development of expertise: The journyy from acclimation to proficiency.
Educational researcher, 32(8):10-14, 2003.
Jacob Andreas. Models of meaning? The 11th Joint Conference on Lexical and Computational Semantics at NAACL, 2022.
Konstantine Arkoudas.
Gpt-4 can't reason. arXiv preprint arXiv:2308.037662 2023.
Emily
Bender.
High resource
languages
vs
low
resource
lan-
guages. The Gradient, 2019.
URL latpss:/thegraddents pub/
the hite-bederrule--- on naming teaming-the-ae Latt--aangaggs--- we- www-stuuy-anddd-ee why -ttyy-tttttttttee.
#fn4
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Kor- bak, and Owain Evans. The reversal curse: Llms trained on"a is f'iail learn"b a". 2023.
URL htLtht:s:://ppp semant www.noicsschlaa wrt. Cttp:/Crrusulic ID: 62083829
David C Berliner. Expertise: The wonder of exemplary performances. Creating powerful thinking in teachers and students, pp. 161-186, 1994.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. Abductive commonsense reasoning. In International Conference on Learning Representations, 2020. URL https:
//openrevicn ew. wwt. www.:oorwwwmmiooyoggoee d= HKDB
10

---

Published as a conference paper at ICLR 2024
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Mare Brysbaert. How many words do we read per minute? a review and meta- eeta-analysis of reading rate. Journal of Memory and Language. 109:104047, 2019. ISSN 0749-596X. doi: https://ooi. org/10. 101.10006....m1200000 104047. URL https::////// //wp: sci encedi com sci ence/
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303. /2712, 2023.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. 2022.
Wikipedia contributors.
Wikipedia:size of wikipedia wikipedia. URL https:
7/en. wikipedia 20of 20oo55002222205pppeemeee22222000
of oat.aaassiiieootiii.pp.aaaaaaaaa http://www...aaaaaaaaa.aaca..aaa
-ext off20al1 altpaaaalssppapass2oooo/o......a
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang. and Ming Zhou. MuTual: A dataset for multi-turn dialogue reasoning. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pp. 1406-1416, Online, July 2020. Association for Computational Linguis- URL https:://ac.aathelooy.. org//020.
tics. doi:
acl mel-main.133
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- ppneral-purpose vision-language models with instruction tuning, 2023.
Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M Ponti, and Siva Reddy. FaithDial: A Faithful Benchmark for iitormationnSeekingg Dialogue. Transactions of the Association for Computational Linguistics, 10: 10:1473- 12 2022. doi: 10.11622tacla..00222
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al. Faith and fate: Limits of transformers on compositionality. arXiv preprint arXiv:2305./880 18654, 2023.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- els. Transactions of the Association for Computational Linguistics, 9:1012-1031, 2021. doi:
10. 00410. URL https::::://acc JRL.attpss//aa..aattooooooooorrp org. 202..
Allyson Ettinger What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34 48, Jan 2020. ISSN 2307-387X. doi: 10.1162/tac_.. 002299 URLhttp://dxx doi. ott./..org/1.... 1162/ tacl 00298
Alvan R Feinstein and Domenic V Cicchetti. High agreement but low kappa: I. the problems of two paradoxes. Journal of clinical epidemiology, 43(6):44,-549, 1990.
Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon. Self-verification improves few-shot clinical information extraction. arXiv preprint arXiv:2306.00024 2023.
Fernand Gobet. Understanding expertise: A multi-disciplinary approach. Bloomsbury Publishing, 2017.
Google. Bard. https::/bard...
com 2023. Accessed before: 2023-09-28.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. arXiv preprint arXiv:2303. 1897, 2023.

---

Published as a conference paper at ICLR 2024
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compeench: A com- prehensive benchmark for open-world compositional text-to text-to-image generation. arXiv preprint arXiv:2307.06330, 2023.
Gabriel Ilbarco. Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi. Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. htRtsstts:://// /pps://do33 .rr.l.rg//// 5281/ zenodo. 5143773
If you use this software, please cite it as below.
Midjoumey Inc. Midjournyy. https::/midjourneet com 2023. Accessed before: 2023-09- 28.
Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. BECEL: Benchmark for consis- tency evaluation of language models. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 3680-3696, Gyeongju. Republic of Korea, October 2022. Inter- national Committee on Computational Linguistics. URL https://Aclaattoooogyy..o
3022.coling-12 1244
Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. Association for Computational Linguistics, 2020.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/w1/D17--0022 URL
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffu- sion model is secretly a zero-shot classifier. arXiv preprint wrxv:22333.16203 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre- training for unified vision-language understanding and generation. In ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint vrXiv:2301.12597 2023b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: Worker and collabo- ration for natural language inference dataset creation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pp. 6826-6847, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.2225 http:///2222...nnsseseea findinggs-emll.55000 URL http::://accaanthoooooooooo
2022. wtd../2......nggneee......
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651 2023.
Microsoft. Bingchat. https://bbltt wtt.ccoccccaaeeeeeee 2023. Accessed before: 2023-09-28.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October-Novenee November 2018. Association for Computational Linguistics. doi: 10.18 URL pattps://aclanthin
OpenAl. ChatGPT: Optimizing language models for dialogue, 2022.
OpenAl. GPT-4 technical report, 2023. URL
12

---

Published as a conference paper at ICLR 2024
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpon natural language processing task solver? arXiv preprint MrXii:2302206476.. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, PP- 8748-8763. PMLR, 2021.
Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. Studying the inductive biases of mns with syn- thetic variations of natural languages. In North American Chapter of the Association for Com- putational Linguistics, 2019. URL httph:ttp///aat wwwant i www.nttccchhhhcccc org/ CorpusID:
80628431
Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Che- ung. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Proceedings of the Ninth Joint Conference on Lexical and Computationnl Semontics, pp. 102, Barcelona, Spain (Online), December 2020. Association for Computational Linguis- org/ 2020. starsem- 1.10
tics. (RLLhttps::2 hattts://aclaathol..o.
Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. Are red roses red? evaluating consistency of question-answeriig models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6174-6184 Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10. 1..18653/11/19-11 1621. URL https: http:////caaathooog...o.ooo org
P19-1 621
Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ro- nan Le Bras, Noah A. Smith, and Yejin Choi. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4661 Online, November 2020. Association for Computational Linguistics. URL ht.t::://///// /ntttss://cll.. anthahooyyyy www.ogg..og.....o.//
doi: 10. Sindings-emnlpp 418
Mihaela Sabin, Karen H Jin, and Adrienne Smith. Oral exams in shift to remote learning. In Proceedings of the 52nd ACM Technical Symposium Computer Science Education, pp. 666- 672, 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494, 2022.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference Em- pirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463-4473, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-4444 1454. URL
Maarten Sap. Ronan LeBras, Daniel Fried, and Yejin Choi. Neural of on the limits of social intelligence in large Ims. arXiv preprint arXiv:2210. 133112 2022.
Minjoon Seo, Tom Kwiatkowski, Ankur P Parikh, Ali Farhadi, and Hannanch Hajishirzi. Phrase- indexed question answering: A new challenge for scalable document comprehension. In EMNLP, 2018.
Gala Stojnić. Kanishk Gandhi, Shannon Yasuda, Brenden M Lake, and Moira R Dillon. Common- sense psychology in human infants and machines. Cognition, 105:00 2023.
Shane Storks, Qiaozi Gao, Yichi Zhang, and Joyce Chai. Tiered reasoning for intuitive physics: To- ward verifiable commonsense language understanding. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pp. 9902-4918. Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 Sindings-emnll.ee antho www.oolloy...gggo// find emn 1p. 422
URL https:: /tatss://accl http://aaaaaaaata//
13

---

Published as a conference paper at ICLR 2024
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. DREAM: A challenge data set and models for dialogue- based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231, 2019. doi: 10.1162/tacl a 00264. URL https:
//aclanthooocc www.ogyy.orrroogmm
Nigar M Shafiq Surameery and Mohammed Y Shakor. Use chat gpt to solve programming bugs. International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455. 5290, (77): 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseeQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4149-4158, Minncapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 11153//11111114211.. URL
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding, In International Conference on Learning Representations, 2019. httpss/ /tpp:://///eeeeeeeee/
net/forum?id=rrJkke.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-ar natural language processing. arXiv preprint 1rXiv:1910.0377/ 2019.
Qingshu Xie. Agree or disagree? a demonstration of an alternative statistic to cohen's kappa for measuring the extent and reliability of agreement between observers. Proceedings of the Federal Committee on Statistical Methodology Research Conference, 4:294, 2013.
Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, and Karthik Narasimhan. COLLIE: Systematic construction of constrained text generation tasks, 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19/ 1472. URL htthtt:::/ www. wwwweb.orgg
anthol anthelloyy/P19-442
14

---

Published as a conference paper at ICLR 2024
A MODELS AND DATASETS
A.1 MODELS
For the language domain, We evaluate the performance of LLMs: GPT4 (gpt-4) (OpenAl 2023
and GPT3.5 (GPT3. 5-turbo) (OpenAl 2022). The evaluations were conducted from July 2023 to September 2023 using the OpenAl API. During inference, we set nucleus sampling to I and temperature to 1. For each task, we evaluate the performance of each model 500 test examples.
For the vision domain, we choose the strongest model available to us (i.e,, Midjourney (Inc.) 2023) the image generator. In practice, Midjourney generates 4 images for each text prompt. For image understanding, we evaluate a wide spectrum of models, including variations of CLIP (Radford et al.
202T), OpenClip (Tharcoo et al. 202T) for selective evaluation, and BLIP (Li al. 2022). BLIP- 2 (Li et al. 2023b). Instruct-BLiP (Dai et al. 2023), Bard (Google 2023 and BingChat (Google)
2023) for interrogative evaluation. For all open-source models, we adopt the implementation and
model weights available on HuggingFace (Wolf et al. 2019).
Given the dialogue history and the knowledge snippet, kindly generate response that faithful the provided knowledge. In this context, "faithful" implies that every piece of information in the response can be verified as true based on the given knowledge.
Knowledge: Waves suitable for surfing are primarily found in the ocean, but can also be found in lakes or in rivers in the form of a standing wave or tidal bore.
Dialogue History:
Speaker A: Sorry to hear you' re terrified from sharks. However, did you know that a surfer is someone who can ride on either the forward or deep face of a wave which typically sends her/him close to the shore? Speaker B: Yeah, what scares is the sharks, is there any non-ocean waves out there?
Response:
Speaker A:
Figure Example prompt for the FaithDial benchmark used for the zero-shot generative setting.
A.2 DATASETS
Language. We examined tasks across five categories: commonsense, NLI, dialogue, summariza- tion. and reading comprehension. For tasks inherently discriminative, where the model chooses from predetermined list of candidates,
omit the construction of negative examples. In gener- ative tasks, we create a list of candidates that includes the groundtruth answer. Specifically, only FaithDial Dziri al. (2022) and XSUM Narayan al. (2018) fall under the generative category, while the remaining benchmarks are designed in discriminative manner. Refer to B.2 for details on negative candidate constructions.
Here, we delve into how we evaluate the dialogue tasks, noting that the same procedure applies to the rest of the tasks. Within the dialogue tasks, our focus spans three benchmarks: DREAM [Sun al. (2019), Mutual+ Cui et al. (2020), where the objective is to generate coherent continuation based on the conversation history, and FaithDial Dziri et al. (2022), where the goal is to produce a faithful response using both the document and the conversation history.
For the generative evaluation in the dialogue task, we input the model with the dialogue history, and if applicable, the knowledge snippet. We instruct the model to generate response that is coherent or faithful, depending on the provided knowledge. For the discriminative evaluation for the same objective, prompt the model to select the correct response from a list of answers, considering the
15

---

Published as a conference paper at ICLR 2024
Given the dialogue history and the knowledge snippet, please select the response from options below knowledge. In this context, "faithful" indicates that every piece of information in the response can be verified as true based on the given knowledge.
faithful the provided
Knowledge: Waves suitable for surting are primarily found in the ocean, but can also be found in lakes or in rivers in the form of a standing tidal bore.
Dialogue History:
Speaker A: Sorry to hear you're re terrifled from sharks. However, did you know that surfer is someone who can ride on either the forward or deep face of wave which typically sends her/him close the shore? Speaker B: Yeah, what scares me is the sharks, is there any non-ocean waves out there?
Option 1:
sure there is. alternative to 1akes and vers which have a phenomenon called standing waves. Have you ever watched people surf?
could be
Option 2: Sure, keep in mind that it is a type of courting You and your girl participate in social activities, but alone, others
don't have to be
be there well.
Option 3: Oh no that is terrible and am sorry to hear that. Option 4: That's 's cool, do you know who the first person to climb it was?
Response:
Figure 8: Example prompt for the FaithDial benchmark used for the zero-shot discriminative setting.
conversation history and knowledge. Please refer Figure for the prompt in the generative setting and Figure 8 for the prompt in the discriminative setting for the FiithDial benchmark.
Vision. For seleetive evaluation, we source text prompts from datasets, COCO (Lin et al. 2014). Paintskill (Cho et al. 2022), DrawBench (Saharia et al.) (022) and T2ICompBench (Huang al., captions on real images. PaintSkill features text prompts that examine image generation specific object categories, object counts and spatial relations be- tween objects. Draw Bench additionally test for long-form text, rare words, and challenging prompts. 12ICompBench is designed test models open-world, compositional text-to-image generation, with text prompts covering 3 categories, attribute binding, object relationships, and complex com- positions. For interrogative evaluation, benchmark. The text prompts in TIFAv1 1.0 are originally from COCO, Paintskill, DrawBench and Parti YYu et al.) (022). For each text prompt, TIFAv1.0 O includes questions automatically generated by language model, only concerning the content specified in the text prompt (e.g., about exis- tence/attributes of an object and relative position between objects).
2023). COCO prompts are
consider TIFAv (Hu al. (023) as the evaluation
B
CAN MODELS DISCRIMINATE WHEN THEY CAN GENERATE?
B.
SETUP
Language.
The basic setup for this evaluation is shown in Figure (l
Vision. We follow the setup on language tasks and consider two settings on each dataset for eval- uation: i) generative: we prompt Midjourney to generate images given the text descriptions, and ii) discriminative: we require the image understanding models to select the image, that better matches the text description, from two candidates. For the generative setting, conduct human evaluations
16

---

Published as a conference paper at ICLR 2024
Selective Evaluation
Interrogative Evaluation
Prompt
Prompt
Q1 Mrite a story about about cherecters baking
Mhat is the capital of Canada?
Understanding
Generation
Understanding &. Washington DC
Story: Once upon time, Tin, Bally, and decided
Generation
0: In the story, did Tim frosting
Ottawa
or after the cake was
Toronto
ataa
A1 rim added frosting after
London
Diagram illustrating the two evaluation settings language modality, selective evalua- tion (left) and interrogative evaluation with the model output each case highlighted orange. In selective evaluation, compare generated responses to selected responses for the same prompt. For interrogative evaluation, we test the ability of models to answer questions about their own gen- erations.
Figure 9:
ChatGPT vs. Human
ChatGPT
Human
Both
1.0
0.8
Generation
0.8
0.6
0.6
0.4
0.4
0.2
ChatGPT
Human
0.0
0.0
CSSA
SociallQA
HellaSwan
PILA
aNLG
FaltDDial
DREAM
Mutual+
WaNU
Nn
KSUM
Toploca
RACE
0.0
0.2
0.4
0.6
0.8
Discrimination
Figure 10: GPT3.5 vs. Humans. Humans show a larger positive correlation between their discrimination and generation abilities compared to GPT3.5.
Figure 11: Quality scores of human generatee re-
sponses vs. GPT3.5 response scores
AMT judge whether the generated image matches the text prompt. In total, randomly sample 100 text prompts per dataset. As Midjourney generates 4 images for each text prompt and users Midjourney practice would pick the best image among the four, we report the success rate per prompt the evaluation metric. For the discriminative setting, we construct the candidates with a negative image for each positive image from the successful generations verified by human workers of given prompt. We report accuracy as the evaluation metric. Human performance on discriminative setting is measured by comparing the majority of human responses to a 4th one.
B.2 NEGATIVE CANDIDATES CONSTRUCTION
Language. To construct the negative examples for the FaithDial and XSUM datasets, we explore two corruptions processes:
we compile responses that are unrelated the information provided in the knowledge snippet K (such as a dialogue or summary document). For a given context, we randomly select a gold response that was based on different K.
Easy negatives:
2. Hard negatives:
To generate examples that are likely hallucinations but sufficiently chal- lenging to distinguish from correct answers, directly perturb the knowledge spans K and then feed them to GPT4, We replace up to two entities in the original K with entities of the same type from the same document avoid casy-tt- off-topic entities. The
17

---

Published as a conference paper at ICLR 2024
sponse generated by GPT4 will be hallucination, containing subtle alterations that render it incorrect when compared to the groundtruth response. For each task, we consider three negative candidates.
Vision. To examine the discriminative performance gap of image understanding models across dif- ferent difficulty levels, we similarly construct hard and easy negatives the image space evaluate image understanding models: i) Hard negative: negative image that is generated based on the same text prompt as the positive image, such that is semantically close to the text prompt, but con- tains subtle mistakes identifiable by humans. ii) Easy negative: negative image that is randomly sampled from the successful generations of a different prompt in the same dataset (as the positive image), such that it is semantically distant from the positive image and can be easily distinguishable. For both cases, we use AMT to verify the negative samples and only retain the ones with agreeable judgments among 3 workers. In the end, we have 345 instances with hard negatives, including 52, 72, 100 and 42 instances for COCO, PaintSkill, CompBench and Draw Bench, respectively; and 372 instances with easy negatives, comprising 82, 72, 100 and 42 instances for COCO, PaintSkill, CompBench and DrawBench, respectively.
B.3 ADDITIONAL RESULTS
Language. In Figure 10 we show humans exhibit larger positive correlation between their dis- crimination and generation abilities compared to GPT3.5. Figure [1 illustrates that workers often favor responses from GPT3.5 over those generated by humans. Figure 12 shows the rationales provided by humans on their preferences for GPT4 responses compared to groundtruth human re- sponses.
Vision. We include additional results from different model variants of CLIP and OpenCLIP in Table These models consistently fall short of human accuracy in discriminative performance. In Table 2 observe the gap between model and human performance becomes larger as the difficulty level of the task increases with easy and hard negatives.
Table 1: Additional results on selective evaluation for vision modality.
COCO
PaintSkill
T2ICompBench
DrawBench
Midjourney (Generative)
85.00%
80.41%
71.15%
62.63%
Discriminative
Human
92.86 %
99.30%
97.00%
100.00 %
CLIP
clip-vit vit base patch11
79.81 %
75.00%
79.00%
16.
clip-vit-base-patchh2
83.66%
72.39%
77.50%
77.38%
cip-vit-llage-pattci. el 1ip=vitt-large-patchh-------
85.58
81.95%
84.50%
78.57%
87.50%
78.47%
81.50%
76.
OpenCLIP
CLi--VE--bigg-11-laioono b160k
81.73%
85.28%
84.50%
84.53%
82.70%
85,41
83.50%
77.38%
83.66%
81.25%
88.00%
79.76%
82.69%
85.41%
83.50%
77.38%
C
CAN MODELS UNDERSTAND WHAT MODELS GENERATE?
The basic setup for this evaluation is shown in Figure (right).
C.I EXPERIMENTAL SETUP.
Language. We additinallyy explore constrained generation in which models are given lexical con- straints for generation. In the constrained setting, compositional task that covers diverse
18

---

Published as a conference paper at ICLR 2024
Table Additional results model vs. human performance across varying levels of answer diffi- culty for vision tasks.
COCO
PainsSkill
T2ICompBench
DrawBench
Hard
Easy
Hard
Easy
Hard
Easy
Hard
Easy
Human
85.71 %
100%
98.61 %
100%
94.00%
100 %
100%
100 %
CLIP
clap-vittbase--aechhe
58.00%
clap-vat-base
67.31%
100%
45.83%
98.95%
55.00%
100%
54.76%
100%
clip-vit-large-patchh
63.89%
100%
19.00%%
100%
75.00%
100%
56.94%
100%
63.00%
100%
52.389
100%
OpenCLIP
bb1 60%
63 46%
100%
70.83%
99.74%
69.00%
100%
69.055
100%
67.31%
100%
62.50%
100%
76.00%
100%:
59.525:
100%
-L2e-ViT-B-10-aa..o
faithdial
WaNLI
better_style
grammatical
longer
sharler
-0.25
0.00
0.25
0.50
0.0
0.5
1.5
CSQA
dream
hallaSwag
better_style
concise
grammatical
informative
longer
shorter
0.0
0.5
1.0
-0.5
0.0
0.5
1.0
1.0
mulual+
PIQA
race
better_ style concise
grammatical
informative
langer
shorter
0.0
0.2
0.4
0.6
1.0
0.5
1.0
sociallQA
topioca
better style concise
grammatical
informative
langer
sharter
-0.5
0.5
0.5
Figure 12: AMT Workers rationals on their preferences for GPT4 responses compared to groundtruth human responses.
19

---

Published as a conference paper at ICLR 2024
Table 3: Example of tasks Collie Benchmark covering several generation levels including word, sentence, paragraph and passage.
Task
Example
word01
Generate word with at least 15 letters.
word02
Generate word with 10 letters, where letter letter 3 'T', letter 'e'.
word03
Generate a word with at most letters and ends with 'r'.
sent01
Please generate sentence with exactly 82 characters. Include whitespace into your character count.
sent02
Generate a sentence with 10 words, where word 3 "soft" and word is "beach" and word 10 is "math".
sentU3
Generate sentence with at least 20 words, and each word less than six characters.
Sent04
Generate a sentence
sure include words "soft", "beach" and "math". Generate a paragraph where each sentence begins with the word "soft".
para01
para02
Generate paragraph with least 4 sentences, but do not the words "the", "and" "of".
para03
Generate paragraph with exactly sentences, each with between 10 and words.
para04
Generate a paragraph with at least 3 sentences, each with at least 15 words.
para05
Generate paragraph with 2 sentences that end "math" and "rock" respectively.
pass01
Generate a passage with paragraphs, each ending in "I sit." and "I cry. respectively.
Paragraph
87
1.0
.5
.0
.12
38
0.5
0.
0.
para01
para02
pera03
para04
para05
Sentence
Passage
1.0
90
0
23
.12
.10
sen1D1
Seni02
Seni03
sent04
pass:01
Word
Cen, Aco.
Diso. Aco. toveral)
85
C5
.5
.15
Disc. Acc, (nnoonect gen)
16
00
wordD1
word02
word03
Figure 13: GPT4 Generative Constraint Satisfaction on Collie along with discriminative accuracy on its Generations.
generation levels, i.e.. word, sentence, paragraph, and passage: COLLIE-v1 (ao al. 2023), which contains 2,080 constraint instances across 13 different task types shown in Appendix Table B We generate outputs for 50 examples per task. We then ask models about their generations, specifically querying about whether the generations satisfy the given constraints.
Vision. For interrogative evaluation on vision modality, we randomly sample 25 prompts from each subset of 1.FAv1.0, resulting 100 prompts in total. For evaluation of image understanding models, we include all answerable questions on the generated images (verified by AMT workers) from the original dataset, and collect the groundtruth answers on this questions from human anno- tators. Note that even when the generated image does not strictly align with the text prompt, we still include the image-question pairs that are considered answerable by human annotators to interrogate understanding models. In the end, gather 1,871 suage-question pairs, with 533, 482, 422 and 434 instances on COCO, Paintskill, BrawBench and Parti subset, respectively. Human performance measured by comparing the majority of 3 human responses and the 4th one.
20

---

Published as a conference paper at ICLR 2024
Interrogative vs Selective Understanding
Selective Understanding
0.85
0.80
0.75
0.70
0.65
0.84
0.86
0.88
0.90
0.92
Interrogative Understanding
Figure 14: A comparison of interrogative and selective understanding in GPT-4 on the 3 tasks they both tested for: XSUM, Mutua++, and HellaSwag. They generally seem correlate, but this will require further study.
C.2 ADDITIONAL RESULTS
Language We report the constrained setting. Figure 13 shows GPT4's constraint satisfaction rate across 13 tasks in Collie.. Certain tasks with simple constraints such as word01 (generating word with a minimum number of letters), and sent (generating sentence containing three spe- cific words) are less challenging for models. However, we observe significant drop in performance when posed with arbitrary position constraints and strict counting requirements (e.g., sent gen- erating sentence with words, where the 3rd word is A, and the 7th word is B, ...). suggesting that current models cannot handle generations when faced with rich compositional constraints. Unlike the open-ended setting, find models often better answering questions about their genera- tions than generating. We propose that more precise constraints like this are easier to judge (trivial for humans) while being much harder for models to exercise flexible generation over.
Further, we compare performance between interrogative and selective understanding for GPT-4 across the tasks they both tested on, finding possible correlation (Figure 14.. This lends some support to the overall difficulty of task understanding being related between different notions of understanding, possibly due to general attributes of the task such as complexity of the underlying text, relationship between input and output, as well as the difficulty of the underlying style of the text.
Vision shows the full results from different model variants of BLIP, BLIP-2, Instruct-BLIP Bard and BingChat on all subsets of TIFAv1.0. Note that Bard and BingChat can occasionally refuse answer the question, when the image contains people. The results from these models on subset when they can provide reasonable answer. The model performance is consistently lower than human performance, acorss different models.
C.3 QUALITATIVE EXAMPLES
Here, we include small-scale study we conducted on GPT-4, of the model's ability to answer questions about stories it generates. Prompts constructed by the paper authors, and questions are constructed by hand to allow probing of specific details the generated content not specifically depending the prompt. We specifically focus simple questions that the model nonetheless gets wrong. These examples are in Tables
Moreover, add more qualitative examples vision modality in addition to what have been shown in Figure() [ Examples of model outputs for selective setting are shown in Figure [5] and those for interrogative setting are in Figure 16
21

---

Published as a conference paper at ICLR 2024
Table 4: More results on interrogative evaluation for vision modality.
COCO
PaintSkill
DrawBench
Parti
Midjourny (Generative)
84.00%
52.00%
72.00%
84.00%
Questiong Answering
Human
95.88%
97.72%
96.32% %
96.83%
blip-vge-base
89.68%
83,82%
82.23%
84,56%
bl4p-wyg--raptit---rrg
89.68%
83.82%
82.23%
84.56%
#1:12-fflan-55--
89.49%
83.20%
81.52%
85.25%
#l.122-flan-t55-xx
91.18%
88.59%
85.31 %
90.56%
BLIP2-opt-2.7D
81.99%
82.16%
72.75%
79.03%
9LIP2-opt-6.7b
84.05%
77.39%
72.99%
75.81% 1%
Lnstructtllpp-flon----11
88.56%
31.54%
81.99%
88.25%
Lnstrucbblip-flan-----x
91.93%
84.02%
84.83%
88.02%
Instructtliip-vycuna--bb
92.50%
$3.20%
81.75%
87.10%
instruchbl1p-vvccua--113
88.74%
80.71%
78.67%
76.04%
Bard
74.02%
66.28%
56.33%
59.42%
BingChat
80.49%
87. 87.20%
80.68%
87.20%
D HUMAN ANNOTATION ON AMT
All human annotations were conducted the Amazon Mechanical Turk (AMT). Through a paid qualification round, we qualify 130 best performing workers that consistently provide conscientious, high-quality annotations. This project paid the Mturk workers between $15-25 per hour in median pay depending on the difficulty of the task. We report on the pairwise agreement rate the agreement levels range from 90-97% over the datasets.
Human Discrimination Evaluation. For the language modality, we obtain human discrimination numbers by prompting the AMT worker with the appropriate context and question for the given task, and ask them to choose the correct response among a list of choices. For vision modality, the set up the same with one exception: the workers are asked to choose the best matching image for the caption. Each examples were annotated by 3 workers and majority vote was taken for the reported results. An example discriminative human evaluation is found in Figure 17
Generation Evaluation. For model generation evaluation in the language modality, the worker is given the context, question, and the model generation and is asked to say if the generation is an acceptable response. the vision modality, the same evaluation is conducted by prompting the worker with a caption (the input prompt) and the generated image and asked if the image matches the caption. Each examples were annotated by 3 workers and majority vote was taken for the reported results. Evaluation of groundtruth also uses this template. Template used is found in Figure 18
Comparative Evaluation. For language modality only, we conduct the comparative evaluation. The worker is prompted with the appropriate context and question, and given model generation and groundtruth answer, asked to choose which the preferred answer. this setup, they are also asked choose from 3-5 checklist that seeks ascertain the rationale for the choice (e.g.. "The response is shorter and so more to the point" or "The style is better"). Template used is found in Figure 19
Human Writing. For vision modality only, we conduct a writing task where the worker is prompted with generated image and question, and asked to briefly answer the question. Each ex- amples were annotated by 2 workers: establish groundtruth answers for generated images based
Our data is skewed to single label. Measures like kappa Krippendorf's alpha are affected by label imbalance and can lower the score even when the agreement high Feinstein & Cicchetti (1990). This makes them unsuitable for interannotator reliability measures in such conditions Xie (2013)
22

---

Published as a conference paper at ICLR 2024
Table Qualitative questioning GPT-4 about own generated stories
Prompt
Write two paragraph story about people building house
GPT-4
Once upon time, in the heart of the countryside, lived three close friends: Bob, architect; Allen, handyman; and Chris, carpenter. They were always dreaming about creating something exceptional and finally, after years of saving money and sketching designs, they decided to build their dream house. Bob, with his architectural expertise, sketched a breathtaking design of wwo-storied country house with wooden porch. Allen collected all the necessary tools, and Chris with his woodworking skills, pledged to make every single piece furniture unique.
Weeks turned into months, and three friends tirelessly worked day and night huild the house. Bob and Chris constructed the wooden framework, walls, and roof; Allen managed the plumbing and electri- cal fittings. During their breaks, they would sit together green hills, discussing the interior decoration and envisaging the fulfilment of their dream. Despite the challenges, disagreements, and exhaustion, they never gave up encouraged one another to keep going. After several months of strenuous efforts, the house was finally built to match their grand blueprint. It was testament to the trio's 's diligent work, creativity, and friendship. and they named it 'The Bond House'.
the raw framework, overlooking the rolling
Question
How many times did Bob sketch the house?
GPTT4
The story mentions that Bob sketched the house once.
Error
We know Bob sketched many designs over the years.
Question
Was the very first sketch of the house two stories?
GPT-4
Yes, the very first sketch of the house was two stories.
Error
We know Bob sketched designs before the first one described, so we actually don't 't know about the very first sketch.
Question
Was plumbing added before or after the walls and roof were constructed?
GPT-4
The story suggests that the plumbing was added after the walls and roof were constructed.
Error
Plumbing typically added before the walls, plumbing inside of the walls.
Question
What did they likely do on the last break before the house was done?
GPT-4
They likely sat together the raw framework, overlooking the rolling green hills, discussing the interior decoration and envisaging the fulfilment of their dream.
Error
The raw framework unlikely be exposed by the time they took their last break.
on majority vote (expert adjudication was used in case of disagreement) and 1 to gauge human performance on the task. Template used is found in Figure 20
23

---

Published as a conference paper at ICLR 2024
A green bus and a yellow tree
A sheep the left of a lamp
Generation
Which image matches the caption?
Which image matches the caption?
Understancling
[selected]
[selected]
photo fire hydrants
plate topped with piece of cake sausage
Generation
Which image matches the caption?
Which image matches the caption?
Understane ganddsg
[selected]
[selected]
Figure 15: Qualitative examples of selective setting on vision modality.
24

---

Published as a conference paper at ICLR 2024
Table 6: Qualitative questioning GPT-4 about own generated stories
Prompt
Write 5 paragraph story about 3 friends finding treasure
GPT-4
Once upon time, in the small, peaceful town of Verdi, three friends- Tom, Max, and Sara lived. Tom was the most adventurous and curious in the group. Max, tallest among them, was most athletic and helped Tom in their ventures. Sara, the most sensible, constantly discouraged their dangerous adventures but couldn't I resist tagging along. One day, Tom, investigating an old myth about hidden treasure, decided persuade his friends join him in his hunt.
The trio cautiously made their way through the dense woods that bordered the town the direction of as stated the legend. Thick webs of strong vines and thomy bushes were a challenge, but Max' strength and Sara's 's whittled down hiking stick hore a path for them. As they reached the clearing at the top of the hill, they saw decrepit stone structure, obscured by overgrown foliage and moss. The friends couldn't contain their excitement Inside crumbling stone structure, there were ancient carvings and paintings along the dusty walls. Although time-worn, they could still make out the drawings. The walls told the story of a mighty king and his beloved, resplendent treasure. Sara, with her keen eyes, discovered hidden passage in the corner of the room. The friends entered the passage with bated breath, their flashlight rays cutting through the darkness. dusty chest lay quietly in musty air, the very image of the treasure chest Tom had described from the legend. Anticipation crackled in the air as they all held their breath while Tom painstakingly opened the heavy lid.
was very likely the location of the treasure according the myth.
The sight that greeted them was beyond their wildest dreams. The chest was filled with gold, silver, and precious gems of every color. There were ornate bands, necklaces, and coins glimmering the torchlight, waiting for centuries be discovered. They looked at each other, their faces mirroring shock and ecstasy of their incredible discovery.
The trio didn't let their triumph consume their logic. They understood such a treasure was part of their town's 's history, and stcaling would be morally wrong. Keeping their excitement at bay, they informed the town authorities about their finding. The town celebrated the discovery, but the greatest joy was in the hearts Tom, Max, and Sara. While they may not have gained matcrialistic wealth, their bond strength- ened, and they discovered the true treasure of friendship and adventure. This adventure was a tale they would reminisce fondly about, well into their old age. The friends garnered respect and admiration from the townsfolk, but they remained humble, attributing their discovery ship had always been cherished, now was legendary. The tale their adventure became a cherished lore, passed down generations, speaking a time long ago, when three friends discovered hidden treasure but chose their friendship and integrity over greed.
luck than skill. Their friend-
Question
Did Sara discourage their adventure to find the treasure?
GPT-4
Yes, Sara initially discouraged their adventure to find the treasure.
Error
The story docs not say this explicitly.
Question
Who would reader expect the trio to inform about the treasure once they found it?
GPT-4
A reader would expect the trio to inform the town authorities about the treasure once they found it.
Error
This is actually surprising part of the story. Generally, reader would expect them to keep the treasure.
Question
What is the most surprising part of the story?
GPT-4
The most surprising part of the story is when the friends, after finding the treasure, decide not to keep it for themselves but instead inform the town authorities about their discovery.
Error
Although this is correct answer, it contradicts what the model responded to the previous question.
25

---

Published as a conference paper at ICLR 2024
Table 7: Qualitative questioning of GPT-4 about its own generated stories
Prompt
Write a paragraph story about 3 scientists building space ship
GPT-4
Once upon a time in the outskirts of London, three brilliant scientists, Dr. Benjamin, Dr. Amelia, and Dr. Max. dreamt of launching spaceship enter the vast, uncharted universe. Their burning ambition and unparalleled intellect dumped them in countless sleepless nights, where they poured blucprints, mathematical equations, and stacks of astronomical data. The trio envisioned building statc-of-the ahe- spaceship powered by revolutionary plasma propulsion technology. Not only would their ship be faster than any mankind had ever built, but it would also have the capability of sustaining life for centuries, hinting the prospect of interstellar travel.
After years of relentless effort and dedication, their dream finally transformed into magnificent spaceship named Exodus. was unlike anything the world had ever seen before. Spanning more than two hun- dred feet in length, the ship looked like cosmic shark decked with arrays of communication antennae and numerous high-tech observational instruments. The generous interior boasted miniature ecosystem, advanced cryogenic facilities, and fully automated systems designed withstand the extremities of deep space. As the world applauded the triumphant spectacle, the three scientists stood smiling, gazing at their incredible creation and dreaming of the extraordinary journeys it would soon embark upon.
Question
Did the trio pour over blueprints, mathematical equations, and stacks of astronomical data before or after deciding to build state-of-the art spaceship?
GPT-4
The trio poured over blueprints, mathematical equations, and stacks of astronomical data before deciding build state-of the-art art spaceship.
Error
The story actually seems imply that they did this because they wanted to build state-of ofte-offthhe-rt space- ship, which would imply happened after.
Question
Why did the trio pour over blueprints, mathematical equations, and stacks of astronomical data?
GPT-4
The trio poured over blueprints, mathematical equations, and stacks of astronomical data because they were planning to build a statc-of-the-art spaceship for interstellar travel.
Error
While correct, this contradicts the previous answer.
Question
Can we say anything about how long the three brilliant scientists knew each other hefore Exodus was built?
GPT-4
The story does not provide information on how long the three brilliant scientists knew each other before Exodus was built.
Error
We can say at least "ycars" because the trio were working together on the ship for years.
26

---

Published as a conference paper at ICLR 2024
baseball game showing batter waiting for pitch.
A blue-colored pizza.
Generation
this a batter?
Is this pizza?
Unders Understanding tanding
Expected
Expected answer:
the image you sent is pizza. pizza typically made a round, flat base leavened wheat-based dough topped with lomatoes, cheese and often various other ingredients. The image you sent shows a rectangular object with brown crust and some white green toppings [...]
not batter. batter mixture flour, fluid, and other ingredients used for cooking, especially for making cakes coating food before frying [...]
Figure 16: Qualitative examples of interrogative setting on vision modality.
Instructions:
Given the context, choose the answer that best answers the question.
Context:
Austin had rough day at work and decided go to the bar. Austin had a drink that night.
Question:
Why did Austin do this?
Choose the best answer to the question.
Scttle down
to the bar
Order drink
Figure 17: Human discrimination template.
27

---

Published as a conference paper at ICLR 2024
Instructions:
context,
question
Document::
Austin had a rough day at work and decided to the bar. Austin had drink that night.
Context:
Why Austin do this?
bar had drink because to relax, unwind, temporarity escape from stress frustrations he experienced during the day. Having a dr nk might provide him with some relief, help him temporariy forget about the workrelated issues, simply serve sodalize and seek solace tought dly..
lkkey
work and wanted
Choose:
Answer Correct the document
Incorrect the document
Figure 18: Model generation evaluation template.
Instructions:
answer to question correct given context,
Context:
Austin had rough work and decided to the bar. Austin had drink that night.
Question:
Why did Austin do this?
Choice Austin likely went to and had a drink because a rough day work and wanted relax, unwind, temporaniyy from the streas frustrations experienced during the temporar ly forget about the work-reloned issues, or simply serve as way to socialze and seek salace after a tough day.
drink might provide him with relict, help him
Choice Settle down
Equally
Choices and are Equally Bed (Or cannot judge because there's something odd going on here)
Why you
option?
apply)
The response longer and so more descriptive
The response shorter and so more to the point
resporse more grammatical.
The langauge better (e use sophisticated words)
The response answers the question concisely without additional irre levnnt information.
response more informative; provides knowledge beyond the asked question.
Figure 19: Comparative evaluation template.
28

---

Published as a conference paper at ICLR 2024
Instructions (clcck expand/collapse)
Thanks participating this HIT!
this you be askec
question about image.
Rules!
1. Answer briefly in few words.
We want you to direct and to the point,
Answer matter-of-factty.
conversational language.
Just state best answer you can give.
Do opinions explanations saying things think.... belleve..
asked and
2. NOT use complete sentences
Instead it kitchen."
"Kitchen".
Instead "Yes "You bet Simply say: "Yes". Instead "No it nott or "Not at all" TNo"
rumerical digits
instead of spe lilling (five, eighteen)
if you speculate g. "What just happened"), provide answer people would agree 5. If you don't know the arswer (e g., specific dog breed), make your best guess.
you really feel like you need to explain yourself on somech ng or point something out to us (say, some weirdness), please do the comments section at the bottom of the HIT.
Please note that these All generated mages. some odd distortions are expected (+.g) person shows fingers instaad of 5). affected by these distortions,
your
briefest way possible. Then point problem
Image:
Question:
this a mountain goat?
Answer brie"y (Did you read the Rules?)
Figure 20: Human writing template
29


Reviews:
[{"review_id": "ukIkVFsqc6", "reviewer": "Reviewer_izog", "summary": "The authors seek to study \"understanding\" in generative models. They conducted several experiments to examine the performance of generative models in both language and image domains in terms of generation vs understanding. It was found that the models surpassed human capabilities in generation tasks as already known but it was found that they consistently under-performed when it comes to understanding.", "questions": "* Did you collect any data which would reveal as to whether the participants can themselves figure if the models don't have understanding? That is, if you let participants interact (perhaps on a task where they need to collaborate) with the model (they are not told that it is a model), what percentage will complain that the model does not understand what it is outputting?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The work is timely, well written and significant for a large section of the conference's audience. \nWhile it is a common observation that generative models seem to struggle with discrimination, this work studies it in a principled way while additionally collecting human data. \nThe work attempts to be complete as it covers a range of tasks, state of the art models and two modalities.", "weaknesses": "* The models tested were trained/finetuned for generation tasks. It is unclear if finetuning for discrimination will fix some issues if not all i.e does the performance drop come from not \"understanding\" or not from being familiar with discrimination tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GPKhL5P7T3", "reviewer": "Reviewer_icxt", "summary": "The paper introduces \"the generative AI paradox,\" highlighting the disparity between generative models and humans, where the former excels in generation but lags in understanding. It substantiates this paradox through extensive experimentation, providing in-depth explanations for the observed phenomena. While \"the generative AI paradox\" resonates with the preconceived beliefs of many researchers, the paper uniquely confirms its validity, offers insightful explanations, and outlines potential avenues for future research, all of which serve as a significant source of inspiration for the broader community.", "questions": "- Could the authors explain which metric more accurately reflects the models' comprehension, nor how each metric contributes to specific aspects of understanding?\n- Could the authors provide a comparative analysis of the generative models' selective and interrogative capabilities, and provide explanations for the observed results?\n- The authors should consider enhancing the clarity and organization of the figures to improve the overall presentation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The generative AI paradox is aligned with the intuitions of researchers, and is important to the community;\n- The conducted experiments are comprehensive.\n- This paper provides some future directions to explore.", "weaknesses": "- The insights presented in this paper may not be particularly surprising to the research community.\n- (Please correct me if I've overlooked any details) The paper employs two metrics, selective and interrogative, to assess the understanding capability of generative models. However, it doesn't delve into which metric more accurately reflects the models' comprehension, nor does it discuss how each metric contributes to specific aspects of understanding. Additionally, a comparative analysis between the selective and interrogative abilities of the generative models is missing.\n- The presentation, particularly the figures, requires further enhancement. Currently, they encapsulate extensive results without a clear presentation way, which somewhat complicates comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GRZimFCtsb", "reviewer": "Reviewer_n1Ya", "summary": "This work identifies and investigates the \"Generative AI Paradox\", where generative models (language and vision) can create outputs equal to (or beyond) that of human experts but do not understand said outputs. The underlying notion is that generation for humans is dependent on understanding (as a prerequisite for expert-level outputs), but the same is not true for generative AI. The hypothesis is tested through an assessment of generative abilities and understanding abilities (split into selective and interrogative settings), for both the language and vision domains. Results show gen. AI outperforms humans on generative tasks but underperform on understanding. The work goes on to discuss the possible reasons for this paradox occurring.", "questions": "1. In relation to the S1 weakness above, are the authors able to suggest any ideas about how the paradox could be overcome? Potential explanations are provided, but how does understanding the paradox enable improved development and a reduction of the disparity between generation and understanding?\n2. The authors note that Figure 2 shows sub-hypothesis 1 is supported for at least one model in 10/13 datasets. Do you have an understanding of why experiments on the other three datasets do not support it?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality**  \nO1. The work demonstrates novelty through the formalisation of hypotheses to capture the paradox.  \nO2. Through evaluation with human participants, novel and concrete findings are established to test the proposed hypotheses.\n\n**Quality**  \nQ1. Evaluation is conducted using several models and datasets for both the vision and language domains.  \nQ2. Thorough analysis shows support for the hypothesis, with a strong accompanying discussion.  \n\n**Clarity**  \nC1. Work is well presented and figures aid understanding.  \nC2. Additional results and further discussions are provided in the Appendix.   \nC3. Paper is easy to follow w.r.t to introducing the paradox, formalising it, presenting results, and then discussing findings.  \n\n**Significance**  \nS1. Further studies beyond initial hypothesis testing reveal additional results, such as human discrimination being more robust to challenging inputs.  \nS2. Some discussion of potential explanations as to why the paradox occurs, e.g. gen. AI trained on generative learning objective - understanding is only encouraged if it furthers this goal (a divergence from human learning).", "weaknesses": "**Clarity**  \nC1. Additional diagrams showing the steps in experiments would aid understanding. This would help highlight the difference between the experiments in Sections 3 and 4, where (from my understanding) the former uses existing candidates to evaluate discriminative understanding, and the latter uses generated outputs. Showing exactly where in the experimental process generative outputs vs existing data are used would be informative. Figure 1 doesn't quite capture the difference between the two cases in my opinion.  \nC2. For the discriminative vs generative subplot in Figure 2, it is difficult to see all of the blue points for GPT4. Adjusting the plot to use alpha values would allow these points to be seen and strengthen the claims supported by the plot. Similar for Figure 8 in the appendix.  \nC3. A handful of concrete outputs would provide further contextualisation for the kind of errors the models are making, as well as give further clarity to the structure of experiments. Figure 1 provides some examples, but additional outputs would be beneficial. Figures 12 to 15 help with this, but the questions/outputs are the same as used in Figure 1.  \n\n**Significance**  \nS1. While the discussion touches on possible explanations for the paradox, it does not mention ways it could be mitigated.  \nS2. The end of the abstract states \"Our findings... call for caution in interpreting artificial intelligence by analogy to human intelligence\". This is briefly discussed under broader implications in Section 7, but I think the findings warrant a more detailed discussion of this outcome and how the results should be used in future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UIRDpVidke", "reviewer": "Reviewer_W7YK", "summary": "Foundation models are growing increasingly powerful. However, their behaviour can be perplexing: often, it seems their ability to generate compelling outputs exceeds their ability to understand their own outputs. These authors put a name to this phenomenon: “the Generative AI Paradox.” The authors attempt to explore dimensions of this phenomenon by comparing human and model performance across two modalities (language and vision).", "questions": "- What data was used for human generations in the language domain? This was not clearly spelled out from my reading? Did you use the language data from the benchmarks discussed? If so, please provide more details.  \n- What version of GPT-4 did the authors use? The March 14 version? The “live” API instance? If the latter, were results conducted all in the same time window? Otherwise, I would worry about a silent update possibly impacting the discrepancy. \n- Can you please provide further details on how participant agreement was calculated? Pairwise in what sense? You reference “kappa” in the Appendix footnote… is this Cohen’s Kappa? Can you provide more details on the skew noted? \n- Minor note which did not impact my score: I would encourage the authors to break Section 2.1 and 2.2 into their own full sections (2 and 3, respectively).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The motivation for the paper is superb. The authors do an excellent job introducing this paradox and it is nice that they put a name to the phenomenon. I am glad to see the authors studying this behaviour. In that sense, the conceptual underpinnings of the paper hold value for the broader ML community. \n\nFurther, I believe that the LLM results are especially compelling. Figure 2 is a nice empirical demonstration of the Paradox. In fact, I was particularly interested in Figure 10 in the Appendix; it is interesting what factors lead human raters to prefer GPT responses over humans. I’d encourage the others to move these findings up to the main text.", "weaknesses": "While I admire the authors’ motivation and their language-centric experiments, I believe that the results for the vision domain are fundamentally flawed. The Paradox hypothesis that the authors raise in Hyp 1 refers to the **same model** being equivalent in generative performance but worse in discriminative performance relative to humans; therefore, I think it is not experimentally sound to have **different** vision models for understanding vs. generation. An adequate investigation into the hypothesis requires having the same model (like the authors did in the language task; e.g., GPT tested on generative and discriminative tasks) rather than one model for generative and (more than one, but different) for discriminative. This discrepancy, I believe, invalidates any of the vision domain results. \n\nHowever, I do not think that this weakness is fatal. I would encourage the authors to focus just on the language domain, unless they procure a vision (or language+vision?) model for which they can study generative and discriminative performance jointly in the same system. If the authors instead focus on just language, I would encourage them to move Fig 10 to the main text (per my note above). At this time, such a chance would be quite major, so I unfortunately recommend rejecting the work. Though, I think the core idea of this paradox, and the language results, warrant further study. \n\nA smaller weakness: the authors are perhaps a bit too flippant about the use of the word “understanding” and “intelligence” from a human cognition perspective. I encourage the authors to look into Gardner’s Theory of Multiple Intelligences in particular. This weakness pales in comparison though to the urgency and weight of the first weakness raised.\n\nMore details on the human data used are needed as well (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)