PoSh: Using Scene Graphs to Guide LLMs-as-a-Judge for Detailed Image Descriptions
Authors: Amith Ananthram
Elias Stengel-Eskin
Lorena A. Bradford
Julia Demarest
Adam Purvis
Keith Krut
Robert Stein
Rina Elster Pantalony
Mohit Bansal
Kathleen McKeown
Venue: ICLR 2026 Poster (2026)
Keywords: detailed image description
metric
benchmark
art
VLMs
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: While vision-language models (VLMs) have advanced into detailed image description, evaluation remains a challenge. Standard metrics (e.g. CIDEr, SPICE) were designed for short texts and tuned to recognize errors that are now uncommon, such as object misidentification. In contrast, long texts require sensitivity to attribute and relation attachments and scores that localize errors to particular text spans. In this work, we introduce PoSh, a metric for detailed image description that uses scene graphs as structured rubrics to guide LLMs-as-a-Judge, producing aggregate scores grounded in fine-grained errors (e.g. mistakes in compositional understanding). PoSh is replicable, interpretable and a better proxy for human raters than existing metrics (including GPT4o-as-a-Judge). To validate PoSh, we introduce a new dataset, DOCENT. This novel benchmark contains artwork, paired with expert-written references, and model-generated descriptions, augmented with granular and coarse judgments of their quality from art history students. Thus, DOCENT enables evaluating both detailed image description metrics and detailed image description itself in a challenging new domain. We show that PoSh achieves stronger correlations (+0.05 Spearman ρ) with the human judgments in DOCENT than the best open-weight alternatives, is robust to image type (using CapArena, an existing dataset of web imagery) and is a capable reward function, outperforming standard supervised fine-tuning. Then, using PoSh, we characterize the performance of open and closed models in describing the paintings, sketches and statues in DOCENT and find that foundation models struggle to achieve full, error-free coverage of images with rich scene dynamics, establishing a demanding new task to gauge VLM progress. Through both PoSh and DOCENT, we hope to enable advances in important areas such as assistive text generation.

Published as a conference paper at ICLR 2026
POSH: USING SCENE GRAPHS TO GUIDE LLMS-AS- A-JUDGE FOR DETAILED IMAGE DESCRIPTIONS
Amith Ananthram", Elias Stengel-Eskin' Lorena A. Bradford Julia Demarest*, Adam Purvis Keith Krut*, Robert Stein Rina Elster Pantalony', Mohit Bansal", Kathleen McKeown
Columbia University "The University of Texas at Austin The National Gallery of Art !UCLA FUNC Chapel Hill ami thêcc col umbi a .dd
ABSTRACT
While vision-language models (VLMs) have advanced into detailed image de- scription, evaluation remains a challenge. Standard metrics (e.g. CIDEr, SPICE) were designed for short texts and tuned to recogmize errors that are now uncom- mon, such as object misidentification. In contrast, long texts require sensitivity to attribute and relation attachments and scores that localize errors to particular text spans. In this work, we introduce POSH, a metric for detailed image de- scription that uses scene graphs as structured rubrics to guide LLMs-as-aJJudge, producing aggregate scores grounded in fine-grainee errors (e.g. mistakes in com- positional understanding). POSH is replicable, interpretable and a better proxy for human raters than existing metrics (including GPT -sa-u-e--- To validate POSH, we introduce a new dataset, DOCENT. This novel benchmark contains artwork, paired with expert-written references, and model-generated descriptions, augmented with granular and coarse judgments of their quality from art history students. Thus, DOCENT enables evaluating both detailed image description metrics and detailed image description itself in a challenging new domain. We show that POSH achieves stronger correlations (+0.05 Spearman p) with the hu- man judgments in DOCENT than the best open-weight alternatives, is robust to image type (using CapArena, an existing dataset of web imagery) and a capa- ble reward function, outperforming standard supervised fine-tuning. Then, using POSH, we characterize the performance of open and closed models in describing the paintings, sketches and statues in DOCENT and find that foundation models struggle to achieve full, error-free coverage images with rich scene dynam- ies, establishing demanding task to gauge VI.M progress. Through both POSH and DOCENT, we hope to enable advances in important areas such as assistive generation. We make our metric and our benchmark available https: /ttp://gttuuu.t com ammtthhhhh ananthram/pan
INTRODUCTION
A picture is worth thousand words can vision-language models (VLMs) capture all of them?
VLMs have saturated traditional image understanding benchmarks from short captioning to question answering (Li et al. 2025). New, more challenging tasks are needed to measure VLM progress. Detailed image description is of particular interest as it requires comprehensive understanding e.g., in Fig. D VLM must correctly specify who is pouring the water. This deep perception is a better proxy for the demands of the real world, where diverse VQA benchmarks (Chen et al. 2024)). Morcover, it enables meaningful applications such as image assistive ("aat"") text generation that could greatly expand accessibility online (Mack al. 2021
queries may not be reflected in
However, making progress on detailed description requires cheap, reliable methods for scoring mod- els. Human evaluation is costly, involving the painstaking comparison of long texts. Even so, there is often no substitute as most metrics were designed for short texts and older models (Berger et al. 2024). Moreover, while metrics that produce a single coarse score of overall quality allow for the

---

Published as a conference paper at ICLR 2026
Description A (better)
BLEU
...In the center stands figure draped in white cloth around the waist, shown in humble, bowed posture. To the right, a muscular figure wearing blue robes holds a vessel and is pouring water. On the left, a figure in vibrant orange....
METEOR
CIDEr
Bert-S
B
SPICE
Description B (worse)
CAPTURE
.wth three figures, all nude. The central figure man who appears to be pouring water from vessel into basin... To his right, there is another man who is seated on rock... On the left side, woman...
LLM-as- a-Judge
PoSh
Figure 1: Failures in attribute/relation attachment common in detailed image description, espe- cially in dynamic scenes. Here, the man pouring water is not central. POSH catches such errors.
ranking of models, they offer little insight into the granular issues driving performance. Granular issues include mistakes in each generation, like the positions of the people in Fig. and omissions in each reference, like the details of the bird's beak Fig. 3 Automatically localizing such errors critical long generations with similar coarse scores may differ in multiple dimensions of interest (e.g., facial features, body orientations, etc.). Otherwise, prompt and/or model iteration necessitates expensive manual inspection understand which description aspects need improvement.
In this work, we propose P a metric for evaluating detailed descriptions that addresses these challenges. POSH extracts scene graphs from generated description and its reference to as structured rubrics for an LLM to granularly identify mistakes and omissions (see Fig. ( pinpointing the textual spans containing errors like attribute/relation mis-hettachmet. Then, it aggregates these localized errors into coarse scores for mistakes, omissions and overall quality. Thus, PoSH weds the strengths of structured methods like scene graphs (Anderson al. 2016), which reduce descriptions to their consequential visual components, with the strengths of L.MMs/VMMs-as-a-- al. 2023), which flexibly compare these visual components against diverse surface realizations.
(Zheng
As PoSH's coarse scores are grounded in its granular scores, it is interpretable, providing clear insights into the errors driving model performance. Moreover, because POSH entirely open- weight, it is inexpensive to use and perfectly replicable, an important pre-requisite for both adoption by researchers and deployment by practitioners that is not afforded by closed models.
Efforts to introduce metrics for longer generations have been constrained by lack of human judg- ments, especially at granular scale and for diverse imagery (see Table [T. To address this, we introduce DOCENT, a novel benchmark whose focus is visual art. DOCENT contains paintings, sketches and sculptures with expert-written assistive text that exhaustively describes features like clothing, physical orientation, relative positioning and gaze, drawn from the U.S. National Gallery of Art (see Figs. and 3p. It includes generations from current VLMs with judgments from art history students their mistakes, omissions and overall quality at two resolutions: granular and coarse. Thus, DOCENT enables evaluating description metrics and deseriptions themselves.
We validate POSH against the human judgments in DOCENT. We show that POSH recovers hu- man description rankings more often (+3 percentage points) and achieves stronger correlations with human-derived scores (+0.05 Spearman p) than existing overlap and open-weight alternatives (e.g. SPICE, CAPTURE, LLa Va-Critic), even surpassing GPT4o-as-a-Judes Moreover, using judgments in CapArena (Cheng et al. 2025). we show this strength is robust image type. Then, given its calibration, we experiment with using POSHI as a reward function for describing the images in DO- CENT and find that this yields meaningfully better descriptions than supervised fine-tuning (SFT).
POSS (PrOofing Scene grapHs) can judge your detailed descriptions what you (really really) want. AI research often uses caption and alt-text interchangeably. However, according to Web Content Accessi- bility Guidelines captions are related to image while alr-text conveys the information in image. As focus evaluating generations that could serve as alt-text, the term description.
2

---

Published as a conference paper at ICLR 2026
Finally, using POSH, characterize the performance of open and closed models in deseribing the artwork DOCENT, establishing a difficult new task. In so doing, we extend detailed description to technically challenging and socially impactful domain: assistive text generation for artwork, whose visual complexity and diversity stress VL.Ms (Bengamra et al. 2024 (see Fig.
In summary, contributions
1. We propose POSH, a new metric for detailed description evaluation. POSH is interpretable, producing coarse scores grounded in granular scores that are localized text spans.
2. We present DOCENT, a new detailed description benchmark with 1, 750 expert-written art descriptions and 900 granular & coarse judgments of generations from informed raters.
3. We show POSH correlates
with DOCENT's judgments than existing metrics and GPT4o while being replicable. On CapArena, we confirm POSH is robust to image type.
4. We demonstrate that using POSH as a reward function outperforms SFT on DOCENT.
5. Using POSH and DOCENT, we evaluate both open and closed models on detailed descrip- tion of artwork, establishing a socially impactful new task to gauge VLM progress.
RELATED WORK
Image description is under-specified the correct way to describe image is often task-specific. This is especially true for assistive text which has context-dependent requirements (Kreiss et al.)
2022). Moreover, in such sensitive applications, correlated failures between reference-free metrics and VLMs relying on similar components could prove dangerous to end users (Deutsch et al. 2022).
Thus, our focus is seference-bbsed evaluation. Traditional metrics were not designed evaluate long
text and can involve truncation due to limited context length (e.g. CLIPScore) (Papineni et al. 2002
Lin 2004 Banerjee & Lavie 2005| Vedantam et al. 2015 See et 2017 Hessel et | 2021 Sarto et 2023). Recent work has explored LLMs/VL.M---s-Jude
though this requires potentially
expensive API calls and offers limited replicability (Chan et al.) 2023 Cheng al. 2025). Even
when replicable, they do not provide interpretable, grounded granular scores (Xiong et al.) 2025)
While prior metrics like SPICE and CAPTURE, leverage scene graphs, they forgo their rich structure by ignoring object attachment (Anderson et al.) 2016, Dong et al.) (024). This favors generations with misattributed details (as in Fig. l). In summarization, Scialom et al. (2021) use question gener- ation and answering (QA) compare summary and its source. In text-to to-image generation, Cho jet al. (2024) use GPT4 to extract and verify a scene graph from a visual prompt. POSH extends these approaches to detailed description evaluation that is replicable and interpretable. With small models, it extracts scene graphs to use as structured rubrics for guiding an open-weight L.MMaas-a-Judge.
Table Detailed image description benchmarks with summaries of their images, reference descrip- tions (where detail is average # of entities + attributes + relations) and judgments (where source is the type of annotator used and time the average time per judgment). Most benchmarks release no human judgments. In contrast, DOCENT contains both granular and coarse judgments of long descriptions of visually complex artwork elicited from annotators knowledgeable in art. Judgments
Images Reference Descriptions
Name
Source Source Words Detail
Source
Type
Time (min)
DCI
web crowd
133
71
DOCCI
web crowd 122
66
no judgments
CompreCap web crowd
DeCapBench uses Imageln Words
IordsInWords
web crowd+
193
113
no judgments with references
DetailCaps
web model
154
95
model
coarse
14.4K
CapArena
uses DOCCI
skilled
coarse
2.4
5.6K
DOCENT
granular
18
300
art
expert
251
161
skilled
(ours)
coarse
5
600
'The judgments in II compare 1) paired references and 2) paired generations for images with refer- ences. As such, they cannot be used to evaluate reference-hased metric.
3

---

Published as a conference paper at ICLR 2026
PoSh
Step Extract
Step 2: QA for Granular Scores
Step Aggregate Coarse Scores
Scene Graphs
generation On the left, a kright... stanci.. In the center, there's a iomen prespes!
Doee the reference describe..
WOTTEIT
wirle
unique entty BAG...
mistakes
"he worran weering blue "woman whit"?
chaliee
io whit.... To the right, another womann. weannn a blue dress Junds golden chalice..
kright
the entiyy [chaliee] as heving the atribule "polierr?
the relution between [hhe woman wearing
blue dross] and (shalce] as "hold"?
kiighht in armor.. strides toward... .. one woman... weaare long long-sioeved. sioeved floor: longth white gown... she booils car yold chalcca.... the second woman looka up and our left. Her sopaz- blue salin crees...
Dons the gencration deserbo...
like..
omissions
woman weering topaz-blue drees"?
"vemnn weering while gown"
knight chalizu
the entyy (lopar-lue drees) as
having the attribute "satin"?
womann
dress
the relition between [women [chalica] as 'hold up"?
white
Figure 2: POSHI, a metric for detailed description evaluation, that produces granular and coarse scores. Step I: Given generated description and its reference, POSH extracts scene graphs that reduce each text's surface diversity to its objects, attributes and relations. Step Using each scene graph as a structured rubric, POSH produces granular scores for the presence of its components in the other text through QA. Step 3: POSH aggregates these granular scores for each scene graph to produce interpretable coarse scores for mistakes and omissions.
Evaluating such a metric requires human judgments of model generations. Though there are many detailed image description benchmarks (Urbanek et al. 2024 Onoe et al. 2024: Garg et al. 2024
al. 2025] Ye al., 2025), most release no such judgments. One notable exception CapArena which contains coarse rankings of descriptions for web imagery (Cheng et al. (025). ). In contrast, our
new dataset, DOCENT contains both granular and coarse judgments, enabling the evaluation of fine-grained metrics like POSH. Moreover, it expands detailed description to artwork whose scene dynamics and expert-writee werttenn references are considerably more complex (see Table
POSH: A NEW METRIC FOR DETAILED IMAGE DESCRIPTION
POSH is a reference- based metric for detailed image description evaluation that takes two descrip- tions, a generation and its reference, and then extracts scene graphs from each to use as structured rubrics for granular and coarse evaluation of mistakes (i.e. precision) and omissions (i.e. recall).
It does so in three steps (Fig. 2): Step 1) It extracts scene graphs from generation and its reference that preserve object attachments. Step 2) It evaluates the presence of generation scene graph com- ponents in the reference (and reference scene graph components in the generation) through question answering with an LLM to identify granular mistakes (and omissions). Step 3) It produces coarse scores for mistakes and omissions grounded in these granular scores. We discuss each step below.
Scene Graph Extraction As in SPICE (Anderson et al. 2016 given a description d, a scene graph G(d) is structured representation of d. Specifically, G(d) (Odd), E(d), K(d) where ((d) C C is set of objects, E(d) C ((d) x A is a set of attributes associated with each object and ((d) C (d) R x (d) are a set of relation edges between objects. C. A and R are open-world sets of all possible object, attribute and relation classes.
Given generation gen with its reference ref we extract sentence-level scene graphs (g(gen), Gr(er) for each using off-the-shelf dependency parsing and combine them via corefer- ence resolution (Honnibal et al. [2200, MMrtinelli al., (024)). This produces scene graphs with full coverage of each text where each component is localized to text spans, allowing for grounded, interpretable scoring. We provide pseudocode for this extraction in Appendix A.2
Granular Scoring Given description d, its scene graph G(d) and different description d', apply the function T to every component e E (dd) to produce a score reflecting its presence d'.
We implement this function via question answering. We produce templated questions for each scene graph component (object, attribute and relation) c E G(d) and prompt an open-weight LLM to quantify the degree to which is described in d'. This avoids forcing an alignment between the
4

---

Published as a conference paper at ICLR 2026
components of G(d) and G(d"). For example, in Fig. the reference describes the figures in the image as a "trio." " Question answering ensures that a generation that refers to all three individually is not penalized for failing to include such collectives.
As objects with the same class may appear many times in a scene graph (c.g., a description of multiple men). questions require the use of unique identifiers (e.g., "woman in white" in Fig. 2 to disambiguate such instances in d'. As the identifier used in d' (if any) is not known a priori, we test candidate identifiers in three passes, first considering only objects not part of any other objects in G(d) (e.g., "man" but not "face of the man"), then objects that are part of other objects in G(d) (e.g., "face of the man") and finally attributes and relations of objects identified as present in d'.
When collecting unique candidate identifiers for an object E O(d), we consider its class name (e.g. "man"), its surface form (e.g. "musician"), its attributes (e.g. "tall man"), its relations (e.g. "man on horse") and if part of previously identified object, its "part-of" relation (e.g. "face of tall man"). We re-write these identifiers using our LLM improve their fluency and then test each one in bulk for their presence in d'. We use the simplest identifier confirmed present by our LLM (if any) to evaluate o's attributes and relations. We provide pseudocode for this templating in Appendix A.1.3
We produce granular mistake scores for every component of Gggen) and granular omission scores P for every component of (ref):
ग (Cgen) V (Cgens ref), VCgen E Gggen)
p(Crrt) V (Crrs gen), Yone G(ref)
Coarse Scoring To maintain interpretabilltyy, we calculate coarse scores for mistakes (i.e. preci- sion) and omissions (i.e. recall) by averaging over our granular scores directly:
Mistakes mean (T)))
Omissions = mean (p(c))
We note this is a natural place to introduce tunable weights (as in [Dogg al. (2024) to adapt POSH to particular datasets. As we aim to demonstrate robustness, we leave these terms unweighted.
4 DOCENT: A NEW BENCHMARK FOR DETAILED DESCRIPTION OF ART
DOCENT is a benchmark for evaluating detailed description metrics and detailed descriptions themselves. It consists of 1, 750 works of art with expert-written references from the Open Data Program at the U.S. National Gallery of Art (NGA fl For 100 of these images, produce four generations from current small and frontier VI.Ms and collect 300 granular (for 75 images) and 600 coarse judgments from annotators knowledgeable in art of mistakes and omissions On average, coarse judgments took 5 minutes and granular judgments took 18 minutes (six annotation days). This highlights both the cost of manual evaluation and the need for metrics that are reliable proxies.
We include summary statistics in Table and example judgments in Fig. B
Image / Reference Selection While the majority of these works are paintings, they include sketches, statues and lithographs (e.g., the bird in Fig. 3) all in the public domain. These im- ages span a diverse set of styles (e.g.. Baroque, Renaissance, Impressionism. Post-Impressionism) themes (e.g., war, courtship, still life, religion) and topics (e.g., fishing, drinking, animals, boating).
The accompanying references are detailed descriptions whose purpose is accessibility - as such, they follow guideline that include tips for deseribing color (e.g.. "color can be likened to temper- ature") and handling ambiguity (c.g, "describe what makes something ambiguous"). These context informed requirements highlight the need for reference based metrics (Kreiss et al. 2022).
Compared to existing detailed image description benchmarks. DOCENT contains considerably more visual complexity (see T). On average, its images contain 16% more objects and nearly twice as many people" who require description of their orientation, features, clothing, etc. Conse- quently, the average length and scene graph size of its reference descriptions are nearly double.
https: / /www nga.nnnna. www.a.gvv/oo
access- mwgee//aaaaaas magps//pannnaaa.ahaiee
We forgo flueney recommended by Kasai et 2022
www. nga. gww/vvsiiiii wwwsis/aacceeeec https/acccsebblcccc bitiiiiiiiyttootoo ct/y//oll 1ecol1cctiio ion- leeettionimageeeeeeeeee iaaeiiooo-meeeeeeeeiiiiooooo
As measured by OneFormer Jain 2023
5

---

Published as a conference paper at ICLR 2026
Granular Judgments
DOCENT III
Reference omissions
bird in flight with its hocked beak wide open, wings held high, holds a shimmering green and white fish its talons this vertical painting. The bird and fish fill the composition against pale. hazy landscape... The bird surges forward, to our left. Its head low...
Generation mistakes
paining portrays a dymamic sceree is positioned in the upper left quadrant of the carvas... The falcon's head turned slightly to the right,, giving the impression movemnnn....
flight. falcon, with wings fully spread,
Coarse Judgments
Generation 1
Generation 2
The painting portrays a dynamic scene of falcon flight. The falcon, with wings fully spread, is positioned in the upper left quadrant the canvas...The falcon's head is tumned slightly to the right...
The painting depicts a majestic hawk in mid-fightt wings fully extended against a light blue sky with wispy white clouds. The hawk's plumage is predominanily white, with dark brown wings and tail feathers featuring lighter brown stripes...
Mistakes: Generation is much better
Omlssions: Gameration 2 is slightly better
Overall: Generation much better
DOCENT, our newly introduced benchmark, is the first to contain both granular (top) and coarse (bottom) judgments from informed raters of detailed descriptions of artwork.
Figure 3:
Model Selection We generate detailed descriptions for 100 images in DOCENT from four cur- rent VLMs that span transparency and model size (from open data/open weight to frontier models): LLaVA-1. 7- (Liu et al., 2024), Molmo-D-7B (Deitke et al. 2024). GP and C1 aude Sennet 3.5. A metric that discriminates among these generations similarly to their human judg- ments could gauge progress detailed image description in small and large VLMs over time. Ad- ditional details (prompts, date of API access) can be found in Appendix A.2.
Annotators Given the complexity of our images and the detail of their expert descriptions, we recruit 24 history undergraduate majors, masters students and PhD students with domain famil- iarity to provide high quality judgments of generations. All annotators were sighted with full color vision and native speakers of English. They were compensated a rate of $22/hour for their time
Granular Judgments Half of annotators identify mistakes and omissions in model gen- erations. For each image, an annotator is shown its reference and then its four model generations in random order. First, they look at the image, read the reference and then the current generation. Next, by selecting narrow text spans, annotators first identify mistakes in the generation (i.e. pre- cision errors) and then omissions in the reference that are in the generation (i.e. recall errors). When identifying omissions, as in Kasai et al. (2022), we ask annotators to mentally correct narrow mistakes in the generation first to avoid double-pente penaaiiin model for both incorrect specificity and lack of specificity. For example, generation that describes a woman as man is an error in precision but not recall. We include our task instructions and interface screenshots in Figs. and 6
Coarse Judgments The other half of our annotators provide coarse judgments of our model gen- erations. For a given image, an annotator is shown its reference and two generations (#1 and #2) in random order and asked to rank the generations in terms of mistakes (i.e. precision), omissions (i.e. recall) and overall quality. These pairwise judgments avoid some of the inter-annotator inconsistency of Likert ratings, especially for long text (Novikova et al. 2018).
Annotators select among five choices for each dimension: #1 much better, #1 slightly better, equal, #2 slightly better and #2 much better. As with our granular judg- ments, we ask annotators to mentally correct narrow mistakes (i.e. precision errors) in each gener- ation before judging omissions. To avoid favoring previously seen generations, we ensure no anno- tator sees the same generation more than once. We include our task instructions and screenshots of our annotation interface" Figs. and
*This study was conducted under Columbia University IRB protocol AAAV6216. Hosted on Label Studio (https: //tas::///ebeee//// 1tt.aaaeeeeee..e.. ic Tkachenko al. 2020-2025

---

Published as a conference paper at ICLR 2026
Agreement For given image, each generation / pair of generations receives at least granular and one coarse judgment respectively. For 15% of our tasks, we collect additional judgments from annotators (2 for coarse, 1 for granular.. Additionally, for 20 granular tasks and 30 coarse tasks, we collect expert judgments from a PhD in art history who authors assistive text at an art museum. We use these extra judgments to calculate agreement in ways (among our annotators and between our annotators and our expert). We report agreement in Tables
Tables4and] and of the Appendix.
For our granular judgments, as recommended by Hripesak & Rothschild (2005) for span annota- tion tasks where the boundaries of negative examples (i.e. non-errors) are ill-defined, we measure agreement using the relaxed F1 (matching spans that contain 50% overlapping tokens). Under this measure, our student annotators exhibit strong agreement among themselves and with our expert.
Our coarse judgments exhibit moderate inter-annotator agreement, with Krippendorf's a = 0.509, 0.409 and 0.459 for mistakes, omissions and overall quality (Landis & Koch 1977). This level of agreement is unsurprising for coarse detailed description evaluation - judgment requires weighing the relative importance of each text's granular enrors and is consequently more subjective. Neverthe- less, our student annotators exhibit moderate to strong correlations with our expert, with significant Pearson P values of 0.727, 0.501 and 0.492 for mistakes, omissions and overall quality respectively.
How well do these VLMs describe art? When considering the performance of the four models included in DOCENT, we observe expected trends, adding to our confidence the quality of our judgments: the smaller models make more mistakes and have more omissions than the larger models (see Tables 4 and 5p. Though most models make few mistakes, they all struggle with omissions. The best model, go covers only 50.1% of the visual information conveyed in DOCENT's references. Raising this requires continued prompt iteration, highlighting the need for an automated metric that can reliably measure both granular and coarse differences in mistakes and omissions.
EXPERIMENTS
POSH We extract sentence-level scene graphs using en_core www.coreeee.bt... wee.trff from Honnibal et al. 2020). transformer trained to perform dependency parsing. To merge objects across these scene graphs while preserving attribute and relation attachments, we use maveri -aress-cc----e-- mes-ontonotes from Martinelli et al. (2024) to perform co co-reference resolution. Our QA scorer V/ is qwen-3-14b Yang et 2025). We template evaluation questions for each scene graph component (as in Fig. 2 re-write candidate identifiers using V to improve fluency and then prompt V to answer each tem- plated presence question by predicting a number between and 5. We extract scores by taking the weighted average over the token logits for each number in Liu et al. (2023). When determin- ing object presence, we use a threshold of 2, determined through tuning on a small hand-annotated validation set. We provide further implementation details and all prompts used in Appendix A.I
Benchmarks
We evaluate POSH against the judgments in DOCENT and CapArena.
DOCENT is our new detailed description benchmark containing judgments from knowledgeable human annotators: granular mistake and omission spans for 300 individual generations and coarse scaled rankings of mistakes, omissions and overall quality of 600 paired generations. We evaluate granular metrics on this benchmark using macro Fl where we credit/penalize a model for predict- ing each annotated/unamootated token. Our coarse judgments are in the form (leath, lerl2, score) where score indicates how much better or worse tezt, is than textz. We evaluate each coarse met- ric m by calculating its 1) pairwise accuracy (whether it picks the better text or a tie, using a tie threshold inferred from the gold tie rate) and 2) Spearman rank p and Kendall's correlations be- tween m (ext)) - m (tertz)) and score, a common practice in machine translation metric evaluation (Kommi et al. 2021) More details can be found in Appendix A.3.1
2025) ) contains 3, 361 images and 10, 348 detailed descriptions generated from 14 current VLMs. 5, 599 pairs of these generations receive coarse judgments from human raters of the better generation (or "tie"). We include CapArena, which contains diverse images drawn from the web, validate metric robustness. However, we note the dramatic simplicity of its images and references compared to those in DOCENT (see Table lp. 64% of its images 1o contain fewer than
CapArena (Cheng et al.
As measured by OneFormer Jain al. 2023

---

Published as a conference paper at ICLR 2026
two objects and 95% depict fewer than two people (compared to 27% and 52% in DOCENT). A metric is evaluated on CapArena at the caption-level (whether it picks the better text or a tie, using a tie threshold inferred from the gold tie rate) and at the model-level (Spearman's 's rank and Kendall's r correlation between ELO rankings derived from metric predictions and gold judgments).
Granular Baselines Our work is the first introduce both metric and a benchmark for gran- ular evaluation of detailed descriptions. As such, this limits our baselines to those able to predict localized mistakes and omissions (i.e., the spans where errors occur). We consider two embedding- based approaches, using Qwen / Qwen 3 Embedding ng 8B from Yang et al. (2025); 4GramEm- bed, which embeds and compares 4-grams from a generation and its reference, and SGEmbed, which embeds and compares components from the scene graphs of generation and its reference. As these approaches (and POSH) produce span scores, we report the maximum Fl scores for mis- takes and omissions across all alerting thresholds. More details can be found in Appendix A.3.3
Coarse Baselines Though POSH is text-only reference- based metric, we select representa- tive set of reference-free (requiring only an image) and reference- (requiring gold standard) pointwise metries (i.e. produce numerical scores) our baselines. These include n-gram overlap metries like BLEU (Papineni et al. 2002), ROUGE-L-Sum (See et al. 2017). METEOR (Banerjee & Lavie 2005) and CIDER /Vedantam al. 2015) and model-based metrics like SPICE (Anderson iet al., 2016), CLIPScore (Hessel et al. 2021) l) and CAPTURE (Dong et al., 2024). Additionally, consider several LLMs/VLMs-as
udde FLEUR (Lee et al. (022), Prometheus (Kim al. 2023), LLaVA-Critic (Xiong et al. 2025). DCScore Ye et al. (2025). Qwen-3 (Yang et al. (2025) and GPT40/GPTS in three settings (reference-free with image, reference- ceasence-based without image and reference-based with image). More details can be found in Appendix A.3.4
Reward Function Finally, given the potential of well-calibrated metric as a verifier reinforce- ment learning (RL), we evaluate POSH as reward function. We train Qwen2. 5.2..5VV.. 5-VL-7B on the 1, 000 images in DOCENT's training set in two settings: 1) supervised fine-tuning (SFT), and 2) RL with DAPO Yu et al. 2025) using POSH. We collect coarse judgments (as in Section 4 for 40 generation pairs from graduate students NL.. More details can be found Appendix A.3.5
6 RESULTS & DISCUSSION
Table 2: Granular metrics evaluated on DOCENT. Reported numbers are the maximum F1 when identifying mistakes and omissions across all alerting thresholds. POSH is best at predicting both mistakes (which are relatively rare) and omissions (which relatively common). As POSH's coarse scores are aggregated from its granular scores, this demonstrates its interpretability.
Random
4GramEmbed
SGEmbed
POSH
Mistakes Fl
0.503
0.483
0.514
0.580
Omissions Fl
0.499
0.641
0.658
0.680
POSH Granular Metric Table 2 presents the performance of POSH and our selected met- rics on identifying the mistakes and omissions in DOCENT. Given the imbalanced nature of our data (where mistakes are infrequent and omissions are common), we report macro averages for each subtask, measuring how well each approach localizes errors within a generation and its reference respectively. First, we note that this task difficult. The considerable room for improvement high- lights the value of a benchmark like DOCENT that contains granular judgments of textual spans. Even so, POSH achieves the highest F1 in mistake (0.580) and omission (0.680) localization. As its coarse scores are aggregated from these granular scores, this demonstrates its interpretability.
CLCLA ATR/Faithscore were not included due to complications with their codebases (Chan al. 2023
Jing
2024.. Due cost (estimated $1, (100), only evaluate DCScore YYe et al. 2025) on DOCENT
8

---

Published as a conference paper at ICLR 2026
Table 3: Selected coarse metrics evaluated on DOCENT and CapArena, identified with + (param- eter count, in billions), _ (requires a reference), (requires an image) and (replicable), "acc" indicates accuracy at predicting the better generation (or "tie") in each judged pair. For DOCENT, p/ indicate the Spearman rank / Kendall's T correlations between differences in the metric and differences in the rank of the generations in each pair. For CapArena, p indicate the Spearman rank / Kendall's I correlations between model ELO rankings derived from metric scores and human judgments. Bold indicates the best replicable metric while underlining indicates the best metric overall. Gray cells indicate correlations that are not statistically significant a 0.05. POSH beats nearly all baselines, including GPT4o, across both benchmarks in all settings (caption ranking of mistakes, omissions and overall quality & model ranking) while remaining perfectly replicable.
DOCENT
CapArrema
Mistakes
Omissions
Overall Quality
Desc
Model
acc
acc
P
acc
acc
P
length
30-5
-0.270
-0.206
37.8
-0.002
0.001
38.0
-0.160
160
-0.121
58.7
0.710
0.582
SPICE
v
41.3
0.308
0.294
55.0
0.464
0.360
58.5
0.458
0.349
41.7
0.275
0.231
CAPTURE
43.3
0.250
0.194
53.8
0.447
0.340
56.0
0.453
0.347
52.5
0.613
0.538
Qwen3
32
57.7
0.282
0.235
53.5
0.256
0.253
61.2
0.289
0:257
56.2
0.890
0.714
Na Criti
72
v
<
62.8
0.412
0.351
57.0
0.509
0.430
86.8
0.546
0.461
64.0
0.987
0.934
DCScore
62.8
0.541
0.422
54.0
0.395
0.298
0.471
0.362
GPT4o
58.5
0,484
0.396
56.0
0,350
0,303
67.3
0.510
014022
55.4
0,880
0,802
GPTS
62.5
0.123
53.2
0.332
0.540
0.410
59.1
0.956
0,846
PoSh
14
<
60.7
0.519
0. 1.105
62.7
9.581
0.451
70.7
0.599
0.166
59.2
0.931
0.796
Table 3 presents the performance of POSII and the best baselines on predicting the coarse judgments in DOCENT and CapArena (full results in Appendix AA.4II).
POSH as Coarse Metric
On DOCENT, across all three dimensions, POSH outperforms every existing replicable metric (i.e.. metrics not reliant on an APD, yielding a 0.11 increase in Spearman p for mistakes (25% f), a 0.07 increase for omissions (14% f) and 0.05 increase for overall quality (9% f) over the next best. It even outperforms GPT4o (in all settings) and text-only GPT5 (on omissions and overall quality). Among all metrics, DCScore (Ye et al. 2025) proves best at predicting mistakes. However, its reliance on GPT4o to extract and verify factoids fails to achieve full coverage of reference detail, underperforming in predicting omissions and overall quality. Despite employing smaller LLM, POSII's 's use of dependency parsing and coreference resolution to extract scene graphs avoids this.
On CapArena, POSH achieves higher caption-level accuracies and model-ranking correlations than nearly every existing weight metric and GPT4o. The sole exception is LLaVa Critic, much larger VLM-as-a-Judee (Xiong et al. (0025). This is driven in part by the simplicity of CapArena (see T.. On the subset of CapArena depicting three or more people (167 judgments), each of whom requires careful description, POSH outperforms LLa Va Critic with model ranking correlations of p 0.727, T 0.581 compared to p = 0.686, 0.550. Thus, POSII robust to image type, excelling in visually complex cases that are of particular interest in detailed image description.
POSH as a Reward Function In Table 7 of the Appendix, we report annotator agreement and aggregate preferences between SFT and DAPO with POSH. each dimension of interest, a POSH- tuned generation earns score between 2 and 2 based on how much worse or better it is than its SFT counterpart. While PoSH-tuned generations had more mistakes (an average score of -0.243), these were incurred in service of much fewer missing details all quality (+0.135). This speaks to the strength of POSH when optimized directly. Given recent progress in generating synthetic detailed descriptions (Li al. 2024), PoSH-tuning could be freely scaled in post-training. Moreover, as POSH produces localized granular scores, it supports token- level guidance (Yang al. 2023), an exciting direction to explore in future work.
resulting in higher over-
POSH Subcomponent Evaluation POSHI relies on two subcomponents, scene graph extraction and scene graph element verification through question answering. As downstream errors may prop- agate from these components, we validate each through comparison against hand annotated exam- ples from DOCENT. We find that PoSH's scene graphs high quality, with average element Fl of 0.892. Similarly, in verifying these scene graph elements, POSH exhibits strong alignment with human raters, with an F1 of 0.852. We provide additional details and analysis in Appendix A.4.2
9

---

Published as a conference paper at ICLR 2026
POSH Runtime A core enabler of POSH'S performance and interpretability is its thorough gran- ular evaluation. It achieves this efficiently through inference optimizations like continuous batching and prefix caching (Kwon al. 2023). POSH scores the 400 examples in DOCENT in 15 minutes, or one every 2 seconds, on a single H100 GPU. In contrast, DCScore (Ye et al. 2025) takes upwards of 2 hours due its heavy use of GPT4 (one description every 25 seconds). As manual evaluation takes 18 minutes per description (see Table []. POSH effectively balances quality and cost.
DOCENT LEADERBOARD
Finally, in Fig. D we plot the POSH scores of VLMs in describing the art in DOCENT. While closed models like Gemini 2.5 Pro lead, open models remain competitive. Improvements will require continued iteration, informed in part by insights gained from analyzing POSH's granular scores.
CONCLUSION
We present POSH, a novel metric for detailed image description that extracts scene graphs to use structured rubrics for guiding LLMs-as-a-Judge, providing interpretable, replicable scores. To validate POSH, we introduce DOCENT, new benchmark with expert-written descriptions of visu- ally complex artwork along with granular and coarse judgments of generations from knowledgeable raters. We show that POSH correlates better than other metrics with these judgments, is robust to image type and is capable reward function. Through POSH and DOCENT, we introduce a leader- board for new challenging task, detailed image description of artwork. It is our hope that this work will drive progress meaningful areas such assistive text generation for artwork and beyond.
LIMITATIONS
Recent efforts have explored using structural priors to guide generation (e.g. Wang al.] (2025) )
As these methods extract and describe structure from images, and as POSH extracts and validates structure from text, we do not expect POSH to be biased towards them in an evaluation setting. Nevertheless, as these models become publicly available, this requires experimental validation.
10 ETHICS STATEMENT
The judgments in DOCENT were collected under IRB protocol AAAV6216 16 with all annotator data anonymized and participants receiving fair compensation (at $22/hour) for their time and expertise.
All of the 1, 750 artwork images in DOCENT are in the public domain, and the expert-written reference deseriptions were published by the U.S. National Gallery of Art under their Open Data Progran specifically for research purposes, ensuring appropriate use and attribution.
While this work aims to benefit accessibility applications for blind and low-vision users, we ac- knowledge that direct community involvement in the development process would strengthen future iterations. However, we note that the expert reference descriptions were written according to the National Gallery of Art's accessibility guidelines which lay out best practices for assistive text.
which
Finally, as with other computer vision systems, this work could theoretically be applied to surveil- lance contexts, but our focus on detailed description does not introduce novel privacy risks be- yond those inherent to existing image analysis technologies. The primary intended applica- tion- improving accessibility- aligns with beneficial societal outcomes.
10

---

Published as a conference paper at ICLR 2026
11 REPRODUCIBILITY STATEMENT
A core motivation behind POSH is improving replicability in detailed image description evaluation through the introduction of performant open-weight metric. In that spirit, we ensure full repro- ducibilty of our findings by:
1. including comprehensive technical details in the Appendix
2. publishing the code for both our metric and our metric evaluations at
com/ ami th-taaaamm anant hramm aarramppore po this implementation supports batch invariance, ensur-
ing perfect reproducibility of our results on an H100 GPU with CUDA 12.8
3. publishing our benchmark at /attps://ggthaii tree/maa wam.amiinddontt...
4. making our models and our benchmark available to the broader research community on HuggingFace
ACKNOWLEDGMENTS
This research is being developed in part with funding from the National Science Foundation under Cooperative Agreement PHY-2229929 (the NSF AI Institute for Artificial and Natural Intelligence) and DRL-21 12635 (the NSF AI Engage Institute), the Columbia Center for Artificial Intelligence and Technology (CAIT) and ONR Grant N00014-23-1-2356 Data and data science collaboration were provided by the National Gallery of Art in Washington, DC. We gratefully acknowledge use of the research computing resources of the Empire AI Consortium, Inc, with support from Empire State Development of the State of New York, the Simons Foundation, and the Secunda Family Foundation. The views, opinions and/or findings expressed are those of the authors and should not be interpreted representing the official views policies of the State of New York, the National Gallery of Art, the National Science Foundation or the U.S. Government.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Siwar Bengamra, Olfa Mzoughi, André Bigand, and Ezzeddine Zagrouba. A comprehensive survey on object detection in visual art: taxonomy and challenge. Multimedia Tools and Applications, 83(5):14677-14670 2024.
Uri Berger, Gabriel Stanovsky, Omri Abend, and Lea Frermann. Surveying the landscape image captioning evaluation: A comprehensive taxonomy and novel ensemble method. arXiv preprint 2xXiv:2408.04900 2024.
David Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, and John Canny. Clair: Evaluating image captions with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13638-13646, 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma. Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen. Caparena: Benchmarking and analyzing detailed image captioning in the Ilm era. arXiv preprint arXiv:2503 /2329, 2025.

---

Published as a conference paper at ICLR 2026
Jaemin Cho, Yushi Hu, Jason Michael Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuse Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine thee-griind evaluation for text-to-image generation. In The Twelfth International Conference Learning Representations, 2024.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salchi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the the- multimodal models. CoRR. 2024.
Daniel Deutsch, Rotem Dror, and Dan Roth. On the limitations of reference-free evaluations of gen- erated text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lunguage Processing, pp. 10960-10977, 2022.
Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Bench- marking and improving detail image caption. arXiv preprint arXiv:2405. 19092, 2024.
Roopal Garg, Andrea Burns, Bureu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Ya- sumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Michael Baldridge, and Radu Soricut. Imageln Words: Unlocking hyper-detailed image descriptions. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 93-127, Miami, Florida, USA, November 2024. As- sociation for Computational Linguisties. doi: 10. (0.18653/v1/2000 http://.2222.2...-mm1.e connp-main.6. URL https: //aclanthologys wwwpolooooo..o.oo///oo 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 7514-7528, 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2019.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial- strength natural language processing in python, 2020.
George Hripesak and Adam S Rothschild. Agreement, the f-measure, and reliability in information retrieval. Journal of the American medical informatics association, 12(3):296-299 2005.
Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 2989-2998, 2023.
Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du Faithscore: Fine-grained evaluations of hallucinations in large vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 5042-5063, 2024.
Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. Transparent human evaluation for image captioning. Marine Carpuat, de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3464-3478, Seattle, United States, July 2022. Associa- h/2022.naacllmain.nnmm
tion for Computational Linguistics. doi: 10. 7/ac //aclanthology.orrg 2022. naacl main.22. 2547
URL https:
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, et al. Prometheus: Inducing fine-grained evalua- tion capability in language models. In The Twelfth International Conference on Learning Repre- sentations, 2023.
Tom Koemi, Christian Federmann, Roman Grundkiewicz, Marcin Dowzyys, Hitokazu Matsushita, and Arul Menezes. To ship not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pp. 478 494, 2021.
12

---

Published as a conference paper at ICLR 2026
Elisa Kreiss, Cynthia Bennett, Shayan Hooshmand, Eric Zelikman, Meredith Ringel Morris, and Christopher Potts. Context matters for image descriptions for accessibility: Challenges for ref- erenceless evaluation metrics. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 4685-4697, 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
J Richard Landis and Gary Koch. The measurement of observer agreement for categorical data. biometrics, pp. 159-174, 1977.
Yebin Lee. Imseong Park, and Myungjoo Kang. Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3732- 3746, 2024.
Xiaotong Li, Fan Zhang. Haiwen Diao, Yueze Wang. Xinlong Wang, and Lingyu Duan. Densefusion-Ine m: Merging vision experts for comprehensive multimodal perception. Advances in Neural Information Processing Systems, 37:18535-18556 2024.
Zongxia Li, Xiyang Wu, Hongyang Du, Huy Nghiem, and Guangyao Shi. Benchmark evalua- tions, applications, and challenges of large vision language models: A survey, arXiv preprint arXiv:2501.02189, 2025.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 71- 2004.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 36296-26300 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511 2511-2522, 2023.
Fan Lu, Wei Wu, Kecheng Zheng, Shuailei Ma, Biao Gong, Jiawei Liu, Wei Zhai, Yang Cao, Yu- jun Shen, and Zheng-Jun Zha. Benchmarking large languaee models via directed scene graph for comprehensive image captioning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19618-19627, 2025.
Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and Leah Findlater. What do we mean by "accessibility research"? a literature survey of accessibility papers in chi and assets from 1994 to 2019. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. pp. 1-18, 2021.
Giuliano Martinelli, Edoardo Barba, and Roberto Navigli. Maverick: Efficient and accurate coref- erence resolution defying recent trends. In Lun-We Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pp. 13380-1333 13394, Bangkok, Thailand, August 2024. As- sociation for Computational Linguistics. doi: 10.18653/v 10118665/v12224..u www2224.allloon...7. URL https: 7/aclanthology.oom org/2044 acl a 722/
Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. Rankme: Reliable human ratings for natural language generation. Proceedings of the 2018 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 72-78, 2018.
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, al. Docci: Descriptions of connected and contrasting images. In European Conference on Computer Vision, pp. 291-309. Springer, 2024.
13

---

Published as a conference paper at ICLR 2026
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http:tttpp.x......o
abs /0s3/198..0 10084
Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Positive- augmented contrastive learning for image and video captioning evaluation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6914-6924 2023.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. Questeval: Summarization asks for fact-based evaluation. In Pro- ceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 6594-6604, 2021.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer- generator networks. In Pruceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 1073-1083 2017.
Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Stu- dio: Data labeling software, 2020-2025. URL https: / ht//ggttuubb../ com/ Open source software available from https///gttub.ccom//uaaaggg////t
studio.
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture worth more than 77 text tokens: Evaluating clip-style models on dense cap- tions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 187000266799 2024.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-baad based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https:///gt httpggttthuuuu....o com www.///wgggggggcc/e////aeeaa//ee
2020.
Jingyi Wang, Jianzhong Ju, Jian Luan, and Zhidong Deng. Llava-sg: Leveraging scene graphs as visual semantic expression in vision-language models. In ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13618 136288, 2025.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. Preference-grounnded token-level guidance for language model fine-tuning. Advances in Neural Information Processing Systems, 36:24466-24496, 2023.
Qinghao Ye, Xianhan Zeng, Fu Li, Chumyuan Li, and Haoqi Fan. Painting with words: Elevating de- tailed image captioning with benchmark and alignment learning. The Thirteenth International Conference on Learning Representations, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
14

---

Published as a conference paper at ICLR 2026
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
A APPENDIX
A.1
POSH
A.1.1
COMPARISON TO DAVIDSONIAN SCENE GRAPH
There several differences between the Davidsonian Scene Graph (DSG) metric (Cho al. 2024) and POSII. While the emphasis of DSG is on evaluating text-to-image models, POSII was designed specifically for detailed image descriptions and is tailored to the unique challenges they pose.
First, the emphasis DSG is on evaluating text-to-image models: it compares an image to scene graph elements extracted from prompt text through visual question answering. As such, it cannot serve as reference-baue metric for evaluating detailed image descriptions. Allowing the use of references is important. Downstream tasks have context-specific requirements that can only be specified with references. This is especially true in accessibility (Deutsch et al. 20222)
Additionally, DSG was designed for image generation prompts at most three sentences long (in con- trast, POSH is able to compare generations and references that are 10 20 sentences long); DSG prompts GPT-3.5 for atomic propositions that are not localized text spans (in contrast, POSH grounds its coarse scores in localized granular scores with open models, allowing error visualization, better interpretability and replicability), DSG's atomic propositions do not have special handling for entity collisions (in contrast, POSH tests discriminating identifiers for colliding entities to allow unique validation of their presence), and finally, DSG applied to detailed image descriptions mea- sures only precision, validating the presence of generation elements in its source image (in contrast, POSH measures both precision and recall, penalizing generations for omitting important details).
These differences in design and purpose become clear when evaluating DSG against the judgments in POSH, where its accuracies are near chance and its correlations are near zero (see Table ().
A.1.2 SCENE GRAPH EXTRACTION
While we provide the complete implementation for our scene graph extraction in our codebase, we include simplified pseudocode below:
def GetGraph (text):
doc ParseTextWWthhhpayyycc (text))
components ExtracttComponents (doc)
corefs GetCooreflittMavrriick (doc)
entities, relations [], []
for each component:
if IsNoun (component): if HasEarlierMention (wr.eeenonnnncoonnneeeeenee (component) UpdateexisttnggEniity
entities, component
else:
CreateNewEntity
entities, component.
for each component:
if IsAdjective (component): UpdateAtributes/ entities, component
15

---

Published as a conference paper at ICLR 2026
elif IsVerb (component) UpdateVerbRelations. relations, component
elif TsPrep (component)
relations, component
return (entities, relations)
A.1.3 GRANULAR QA TEMPLATING
While we provide the complete implementation for our question templating in our codebase, we include simplified pseudocode below:
def cemplaatenniityuusstiiin text, entities
colls Getcollisions (
entities
questions for e in entities:
identifiers []
if (sEmpty(colls)
identifiers. add (e. text)
for each attr in e:
if IsUnicue (attr, colls): identifiers. attr e.text
if len (identifiers) > 0:
AddToquestions (identifiers) continue
for each rel in if TsUnique (rel, colls)
rel.head +
rel.text +
rel.tail
AddIoQuestions (identifiers)
ReRtiteeddenifiierss (questions)
def TemplatesttrReeqqeetiion
text, entities
questions [] for e in entities: for attr in e: AddioQueations attr, e.identifier
16

---

Published as a conference paper at ICLR 2026
for rel in e: AddIoQuestions rel, e. e.identifies
A.1.4 PROMPTS
Entity Identifier Rewrite Prompt (for attributes)
Rewrite "(entity identifier }r into grammatically correct noun phrase, keeping all details. For example, ddog small" should be rewritten as "the small dog". Output ONLY the phrase
Entity Identifier Rewrite Prompt (for relations)
Rewrite Seentity_ieettiferr }r' into a grammatically correct noun phrase. keeping all details. " "cat jumps on window" should be rewritten as ''tee cat jumping the window". Output ONLY the phrase.
Verification Prompt
if (precision) DESCRIPTIONI: {target-text]
DESCRIPTION2: (source text]
{else]
DESCRIPTION: {target_text}
{if entity)
Q: Is an entity matching (from DESCRIPTION2) mentioned in (the) DESCRIPTION
{elif attribute]
Q: Is ventity_identifier) (from DESCRIPTION2) described as "(attribute)" in (the) DESCRIPTION (1)?
{else}
identidierr Q: Is the relation between ideniiiilieenniiirrrrr }" and "fentity2.identifii described "[relation 1" in (the) DESCRIPTION (1)?
(in DESCRIPTION2)
Consider paraphrases but do NOT infer unstated details.
Scoring guide -> 1: absent: 2: weak hint; partial; 4: clear; 5: explicit & unambiguous.
Respond ONLY with an integer 1-5.
A.2 DOCENT
A.2.1 GENERATIONS
We produce generations from the following models:
17

---

Published as a conference paper at ICLR 2026
1. -aava-vv.--nnistzaa1--------- on HuggingFace (Liu et al. 2024
2. Molmo-7B-D-0924 on HuggingFace (Deitke et al. 2024
3.
06, accessed on 1/31/25
4. claude --n-eenett 202410224 accessed on 1/31/25
We use the same prompt (included below). For LLaVA-1 7B and Mo 1mo---7B, we use nu- cleus sampling Holtzman et al. (2019) with 0.9 and a temperature of 0.7.
Detailed Description Prompt
[IMAGE]
Generate a detailed description of this painting, avoiding interpretation and focusing only its visual elements.
A.2.2 AVOIDING DOUBLE PENALTIES
In Kasai et al. (2022), after identifying error in precision, the authors correct the error before annotating recall. This avoids doubly penalizing description for errors in specificity which would unfairly favor more generic descriptions (which only penalized once, for recall). We instruct our annotators to do the same.
Due to the length of the generations and descriptions in DOCENT, please consult codebase for example judgments: htt.::::tt/s/..... /tttts://githbb.. cctm://mthhaaaaaarrraaaaaaaareea docent pecent/exanne wttee/eeaeegggaall./.
18

---

Published as a conference paper at ICLR 2026
Granular Evaluation Image Descriptions
Hello, thanks for being part of our research study. Our goal is to accurately characterize the performance of vision-language models (i.e., Al systems that can describe images). By doing so, you'll help gauge how well such systems would perform in consequential settings such as the automatic generation of accessibility text people who blind
low vision.
our annotation interface, you'll see 1) an image, 2) a CORRECT description the image and 3) a GENERATED description of the image. Your task is to first identify minimal spans in the GENERATED description that are mistakes (e.g. incorrectly added details that are not true the image: identifications of nouns, their descriptors their relationships that not true of the image) and then identify minimal spans in the CORRECT description that are missing (e.g. details not reflected in the GENERATED description). For each task, please follow the instructions below:
1) Look at the image. a quick sense any relevant people or objects, their actions their broader setting.
2) Read the CORRECT description of the image.
3) Read the GENERATED description of the image.
4) Read GENERATED description image again. you encounter mistakes (e.g., incorrectly added details that are not true of the image or nouns, their descriptors or their
Figure 4: The beginning of our granular annotation instructions, also hosted on our GitHub
Evaluation of Image Descriptions
Hello, thanks for being part of our research study. Our goal is accurately characterize the performance of vision-language models (i.e., Al systems that can describe images). By doing so, you'll 'lll help us gauge how well such systems would perform consequential settings such automatic generation of accessibility text for people who are blind or have low vision.
In our annotation interface, you'll see 1) an image, 2) CORRECT description of the image and 3) two GENERATED descriptions the image. Your task is to provide relative grades of the GENERATED descriptions across three dimensions: mistakes, missing details and overall quality.
Mistakes in GENERATED descriptions incorrectly added details identifications of nouns, their descriptors or their relationships that are not true the image. Missing detalls are details in the CORRECT description that are accounted for in the GENERATED descriptions after correcting their mistakes. Overall quality is more subjective we want you to grade the generations by which one is the best stand-in the CORRECT description.
For each task, please follow the instructions below:
1) Look the image. Get a quick sense any relevant people or objects, their actions and their broader setting...
Figure 5: The beginning of our coarse annotation instructions, also hosted on our GitHub
19

---

Published as a conference paper at ICLR 2026
Desclptiin
ecom androse aiee grea fover help plener ng Blery siet wi The SHE shance
ihos nar the setie conta the conocittio dradee you oie to eech dis booe the ren briind Tovaritg iely oiee Thoer
mest op
dws Jervice cor wif The selion cones absuc thoou the the so corppcaioo and the pare sscce mntoe uer jur. sivee enterin comer CONOT'
with couce
The puna coaii sosa my munk thes wat aiggen Eiues oegi the seac cook witn entretie
verged seppaing
bross whi acdractars vabbe hrrer lemans na
twe't 'slye denas and uss with the ne encinn viriey mooe aus incor parding vay deneir inppessioo aedi sryl whone
in deyi pereene leceal
marrggu tita noor ove
stng with precee ua gierr dives
vallo saluue mospaton eative
suubr sour
dital
Figure 6: Our granular annotation interface, hosted on Label Studio (https://labbe www.llaleelllaeee.. ii))
GENERATED Dasc rptioon
GENERATED Descripionn 92
ons ty
shous
oten oct toeing of olthere veeders has thove
men signpp un soorle dercctioos
conge lieure was corvent sakion salirmo toree
induiin
Den helher "ircs"
verged vodorm
risting
wdins thoead
on
aoreesioe neeeee The oiiing thee fiueest
gosent
secgpouee mreve thee dy with unde chuut the right the there come
Desoription
mann chet accabos in paattey vi tna Alt vor spean to car noctto
wfree somee
ser
thrsegy
repoast donel veryy
tgans set uir ctalin sicter spice culial 5000 atta comnvite
rye man' cta't creen Boos
tynets
gertt
cartal man' Vor and seril chiocc out
the
Maring Fenalt conecting minso
courder versmecs the with toears ceetiige
tlack undea valinci anrik
mayy
ropers trannces and smooth tes
Figure 7: Our coarse annotation interface, hosted on Label Studio /https:// lstud. 1o).
20

---

Published as a conference paper at ICLR 2026
A.2.3 DOCENT: AGREEMENT AND JUDGMENT SUMMARY
Table 4: Granular judgments of mistakes and omissions in DOCENT. Left: inter-annotator agree- ment (relaxed FI, using overlap thresholds of 1 token and 2 50% tokens); additionally, the recall R of our expert annotations. Our student judgments exhibit strong inter-annotator agreement and good coverage of our more sparing (see Fig. ( error annotations. Right: the average percentage of tokens not marked as mistakes/omiissios for each model (higher is better).
student
expert
llava
molmo
claude
gpt4o
F1
Fl @50
R
F1
R@50
Fl @50
mistakes
0.980
0.604
1.000
0.890
0.652
0.250
0.886
0.920
0.961
0.957
omissions
1.000
0.754
1.000
1.000
0.927
0.475
0.359
0.468
0.462
0.501
Table Coarse judgments precision, recall and overall quality in DOCENT. Top: inter-annotator agreement (Krippendorff for student, Pearson and average difference for expert). Bottom: the average relative score of each row model compared to each column model (1 indicates the row is much better, 5, the column is much better).
mistakes
omissions
overall quality
student (ax)
0.509
0.409
0.459 0.492, 0.788
expert (p.
0.727. 0.633
0.501, 0.644
llava
molmo
claude
llava
molmo
claude
llava
molmo
claude
molmo claude
2.42
2.32
2.14
1.92
2.37
2.16
2.74
1.87
2.5
gpt4o
1.86
2.3
3.0
2.01
2.54
2.61
1.68
2.21
2.63
A.2.4 DOCENT: GRANULAR AGREEMENT DETAILS
We additionally calculate granular agreement using a more conservative threshold (2 50% token overlap). Here, relaxed Fl remains strong among art history student annotators (0.612 for mis- takes, 0.773 for omissions). Though we observe drops in relaxed Fl when compared to our expert, it is driven by two factors: annotation style, with our expert favoring sparsity, and relative strict- ness on the part of our student annotators. This reflected in the expert annotation recall values in Table 4 where majority of the spans identified by expert were also marked by student an- notations for both mistakes (0.652) and omissions (0.927). Thus, our expert annotations are a subset of our stricter student annotations. We provide side-by-side example of student annotation and an expert annotation in Fig.
21

---

Published as a conference paper at ICLR 2026
the com oot wae four
intftcctuna doments and fon deoppe her brrung
the carnec two fii are flewing rober one vesting sright the and other brownd show greating
these fipur the strese th and
beards
iolea
ond
locu the enry sovctaciess has out hame near rain ne using whstt sppears be wattong anis.
bacigg puad shovc dets chcape with acchnsctanii amartts tovart bukidn on boct oldes caling
trt canttu stwnton to desei the rener of hebe withiihh distorato taide Thi qustiis yyur varuaaly snleaae, wi the two cle ding tige in the dee bar unstier either
pugest comporet glou
Student Judgment of Omissions
An Expert Judgment of Omissions
CONNECT Description
CONNECT Desoription
Sooon revil gafieee the lomanigg, and jommal custeri front builing kinokel
the cinatl have polo sverty Mill Olver to the right df lowiede bus
Riscossid pelll
the puppo boe biki avettyase Cose un the both Butniieent cake tettaiis They sand with thet
burnieeet plate HMl0 They
umpone
bondhai buist
eveer Birret hill which
insidess Lavning towart wer co soannon Hey joss disccl we the ees of 3600 norves Eiibees They Biep
recuter lieeiig comornee Mery paer divetty Restiicees of
women Chesk They ring
gantrnnk ane unsaiata mosiasi stoe
only
sacgracht a
Stathen Croaathh incutii stave ropemne enty shrer teclopoand fendecass
bpoak dispet
chell lntter
rewe siin diceer has tns bomeeyy
pere
mag a0 diginty risas auctare the
wine sas Eslin aighl toearte
He's The and signeyy vavee stadioe the edeery of oihee rode thit Booiss argied tnsciii the
walk massing coe Biiling Frank Rigpe paa B00 pay
fars twnp
boat over beling wass
pereest gou ong pray
dless
farthni book to
red to hett On our git second mar vieis Black GAcl epecaces procedoee witk The seacurni
Ill rest fin, newr Ho our record ana weer back vourd specac ss parched th noee. hood
allong was
shoakees en ahive grue rape
corlin
foy
Angs 186 gamatt and Boold on the aore flor betweer the mans Aeyond the for people the sar coreeed and
din devm retwere beyn ou resk
dis down
right buiding Booena
tasle onagel aganaal wel
holdi wolk
chitt
creaingg or rgii angal holding oith belty tonard monen
ihaed and crisel and cheit.
To tie paeto cnt
Deond ostend body
Figure 8: A comparison of our student and expert judgments of omissions for the same genera- tion. Most differences are due to 1) students preferring the specificity of terms like "women" over "figures" and 2) students annotating all the attributes and relations of entities marked as missing, e.g., "skin", "halos", "noses", and the span beginning "painted with a scene..." ... Generally, expert judgments are a subset of our student judgments for these reasons.
22

---

Published as a conference paper at ICLR 2026
A.3 EVALUATION
A.3.1 METRICS
Spearman's Rank Correlation Coefficient (p) assesses the monotonic relationship by calculat- ing Pearson's s correlation on the ranks two continuous variables rather than their raw values. ranges from 1 to +1, with +1 indicating perfect monotonic increasing relationship and -1 indi- cating perfect monotonic decreasing relationship. It's less sensitive to outliers than Pearson's 's and can detect monotonic non-linear relationships. As the coarse annotations in DOCENT specify the rank of two generated image descriptions, Spearman is well suited for evaluating
Kendall's measures the ordinal association between two variables based the ranks of the data. It ranges from -1 to +1, where +1 indicates perfect agreement between the two rankings, indicates association, and -1 indicates perfect disagreement. Unlike Pearson's, ss Kendall's 's tau is non-parametric and robust to outliers, making it appropriate for non non-linear relationships and non- normally distributed data.
A.3.2 COARSE SCORE SCALED EVALUATION
We convert each coarse judgment of generation pair (text,, text2, label) in DOCENT to numer- ical score s that reflects the relative rank of text; and text2. text, was marked as much better and equal to text2,
than text2, 2; s slightly better than text2, text, was marked as s i iigll better than text,, 8 = -1 and 8 = -2 if much better than evaluate
0. Similarly, if
text These numerical scores reflect the relative rank of text, and text, and allow the correlation of different metrics m with the coarse judgments in DOCENT by comparing 8 to m(text]) mtexxt2) with appropriate measures of monotonicity like Spearman's rank correlation p.
A.3.3 GRANULAR BASELINES
4GramEmbed We extract all of the 4-grams from each sentence of generation and its refer- ence, embed them using Qwen/Qwentt Twwen//qee3-Embeeee - (Reimers Gurevych] 2019 Yang et al. 2025) and then calculate the maximum pairwise similarities between generation 4-grams and ref- erence 4-grams. Generation text spans and reference text spans with maximum pairwise similarity scores lower than 0.7 were predicted mistakes and omissions respectively, threshold chosen to maximize the macro F1 scores reported for 4Gram
Table 2
SGEmbed We extract all of the components (objects, attribute-object pairs, and object-relation- object triples) from the scene graphs of a generation and its reference extracted for POSH in Sec- tion embed them using Owen/
(Reimers & Gurevych 2019 Yang
et al. 2025) and then calculate the maximum pairwise similarities between the generation com- ponents and the reference components. Generation components and reference components with maximum pairwise similarity scores lower than 0.8 were predicted as mistakes and omissions re- spectively, threshold chosen maximize the macro Fl scores reported for SGEmbed in Table 2
A.3.4 COARSE BASELINES
When prompting GPT4o and GPTS to evaluate our generated detailed image descriptions, use three different prompts depending on whether we are including the image (reference free) or including the reference. Additionally, a detailed scoring rubric for each score type (mistakes, omissions and overall quality) though we find that this setting underperforms the simpler prompts below.
experiment with more complicated prompt that includes
10-p-40-2222 20000-02---- and aptt5-2025-085 (with minimal reasoning) accessed 9/17/2025
23

---

Published as a conference paper at ICLR 2026
Image Only
[IMAGE]
Generated Detailed Description: [GENERATION]
Please provide numerical scores (from 0 to 5) for the precision (e.g. mistakes in the generated description), recall (e.g. missing details from the image). and overall quality of the generated detailed description compared the image. Output your answer as a JSON dictionary with the keys 'precision 'recall" and quuaity
Reference Only
Reference Detailed Description: [REFERENCE]
Generated Detailed Description: [GENERATION]
Please provide numerical scores (from 0 to 5) for the precision (e.g. mistakes in the generated description), recall (e.g. missing details from the reference description), and overall quality of the generated description compared to the reference description.. Output your answer as a ISON dictionary with the keys 'precision" 'recall and "overall.qualigg
Image & Reference
[IMAGE]
Reference Detailed Description: [REFERENCE]
Generated Detailed Description: [GENERATION]
Please provide numerical scores (from 0 5) for the precision (e.g. mistakes in the generated description), recall (e.g. missing details from the reference description), and overall quality of the generated detailed description compared to the image and the reference description. Output your answer as a JSON dictionary with the keys 'precision" 'recall ", and overalll
A.3.5 REINFORCEMENT LEARNING
We train Qwen2. 5 1--V-77 7B on the 1, 000 images in DOCENT's training set in two settings:
supervised fine-tuning (SFT) with full parameter updates using a learning rate le - 5, linear warmup ratio of 0.1, and effective batch size of 64 for epochs, choosing the checkpoint with the lowest loss on DOCENT's validation set
2. DAPO Yu et al. 2025) with full parameter updates, implemented with TRL (von Werra et al. 2020). using learning rate of le 6, 20 warmup steps. generations per sample (with a temperature of 1.0 and top, = 0.7), = 0.28, B = 0, and an effective batch size of 64 for single epoch, choosing the final checkpoint
24

---

Published as a conference paper at ICLR 2026
We ask seven graduate students in NLP to compare and evaluate SFT and DAPO generations (greedily sampled) for 40 images from DOCENT's test set. Additionally, we collect three annota- tions for five of these images to calculate agreement.
A.4 RESULTS
A.4.1 COARSE
Table 6: All coarse metrics evaluated on DOCENT and CapArena, identified with è (parameter count, in billions), (requires reference), (requires an image) and (replicable). "acc" indicates accuracy at predicting the better generation (or "tie") in each judged pair. For DOCENT, / indicate the Spearman rank / Kendall's т correlations between differences in the metric and differences in the rank of the generations in each pair. For CapArena, p indicate the Spearman rank / Kendall's correlations between model ELO rankings derived from metric scores and human judgments. Bold indicates the best replicable metric while underlining indicates the best metric overall. Gray cells indicate correlations that are not statistically significant at 0.05. POSH beats all replicable baselines and GPT4o on DOCENT in all settings (mistakes, omissions and overall quality) while remaining perfectly replicable. Moreover, POSH is robust, achieving the second best score among replicable metrics on CapArena.
DOCENT
CapArena
Mistakes
Omisstions
Overall Quality
Desc ace
Model
ace
ace
ace
P
longth
30.5
-0,270
-0.206
37.8
-0.002
-0.001
38.0
-0.160
-0.121
58.7
0.710
0.582
BLEU-4
31.2
-0.070
-0.053
12.5
0.118
0.087
12.8
0.051
0.038
17.4
0.421
0.319
CIDER
32.0
-0.118
-0.050
37.5
-0.009
-0.007
37.8
-0.106
-0.079
38.4
-0.279
-0.209
METEOR
16.2
0.197
37.6
ROUGE-LS
37.5
0.251
0.190
44.0
0.214
0.161
47.3
0.210
0.158
45.8
0.180
0.199
SPICE
41.3
0.308
0.234
55.0
0.464
0.360
58.5
0.458
0.349
41.7
0.275
0.231
CAPTURE
43.3
0.259
0.191
53.8
0.141
0.340
56.0
0.453
0.347
52.5
0.613
0.538
CLIPScore
45.3
0.145
0,108
47.0
0.176
0.133
$3.5
0.181
0.136
32.5
0..574
0.451
38.5
Prometheus
9x7
51.2
0.014
0.011
49.8
0.136
0.116
58.5
0.007
0.007
53.9
0.859
0.648
Owen3
32
57.7
0,252
0.235
51.5
0.256
0.253
61.2
0.289
0.257
56.2
0.889
0.714
Va Critic
62.8
0.412
0.351
57.0
0.509
0.430
65.8
0.546
0.461
64.0
0.
0.934
DSG
41.3
0.081
0.068
37.3
0.033
0,0024
45.3
0.017
0.012
DCScore
'
54.0
4PT4n
63.2
0.489
0.400
55.5
0.338
0.274
68.7
0.477
0.301
53.6
0.868
0.602
4PT-o
/
53.3
0.324
0.261
50.0
0.277
0.215
60.8
0.388
0.297
56.77
0.867
0.683
58.5
0.484
0.396
56.0
0.380
0.303
0.510
0.402
55.4
0.890
GPTS
66.0
0.581
0.176
55.5
0.151
0.315
69.2
0.593
0.166
56.9
0.916
0.802
GPTS
62.5
0.511
0.423
51.2
0.421
0.332
68.0
0.544
0.440
59.1
0.956
0.846
GPTS
68.2
0.601
0.191
56.3
0.177
0.366
67.2
0.602
0.175
62.1
0.931
0.816
POSH
14
60.7
0.519
0.405
62.7
0.581
0.451
70.7
0.599
0.466
50.2
0.901
0.796
Table 7: Annotator agreement (Krippendorff's a') and aggregate preferences between variants of Qwen2.5- VL-77B trained on DOCENT: tuned with DAPO using POSH ("PoSH-tuned") and super- vised fine-tuned ("SFT"). A PoSh-tuned generation earns score between -2 and 2 based how much worse or better it is than its SFT counterpart. Reported numbers are averages of these scores.
Mistakes
Omissions
Overall Quality
αx
0.235
0.464
0.184
PoSH-tuned vs SFT
-0.243
0.432
0.135
A.4.2 SUBCOMPONENT PERFORMANCE
To evaluate the scene graph extraction subcomponent of POSII, we hand annotate scene graphs for ten descriptions in DOCENT, five references and five generations. We measure precision, recall and Fl for entities, attributes and relations and, for each matched element pair (i.e. entity, attribute or relation), accuracies for coreference resolution, attribute attachment and relation head and tail attachment. The numbers reported in Table are averaged across the ten descriptions.
25

---

Published as a conference paper at ICLR 2026
To evaluate the scene graph element verification subcomponent of POSH, manually answer 620 templated questions for two randomly sampled inveration-referene pairs in DOCENT and com- pare them to POSH's presence scores. The numbers reported in Table are are the maximum achievable Fl scores across all alerting thresholds. The errors in this subcomponent stem from generations specifying correct details not present in the reference, relative permissiveness toward interpretive language in generations and limitations of the POSH's templating and unique identifier discovery logic. Nevertheless, the scores speak the strength the POSH framework and the potential of future subcomponent improvements to yield further gains in performance.
Table 8: Evaluation of PoSiI's extracted scene graphs for ten descriptions in DOCENT. "P" indi- cates precision, "R" indicates recall, "coref" indicates accuracy at predicting each matched entity's 's first mention. "head" indicates accuracy at predicting each matched attribute or relation' ' entity sub- ject and "tail" indicates accuracy at predicting each matched relation's 's entity object.
Entities
Attributes
Relations
F1
coref
R
F1
head
R
F1
head
tail
0.937 0.883 0.909 0.925 0.881
0.917
0.898
0.951 0.978 0.794
0.872
0.850
0.926
Evaluation of PoSH's scene graph verification for 620 elements from two generation- reference pairs DOCENT, broken out by mistakes (verifying generation scene graph elements in a reference) and omissions (verifying reference scene graph elements in generation).
Table 9:
Mistakes F1
Omissions F1
Overall F1
0.941
0.754
0.852
VLM Performance on DOCENT
Open Models Mitzaees
Open Modelss Omiistons
Open Models Oweral Quality
4.000
EX0
4.500
Boen
640
C3
p0100
paisedend
Closed Modeds- Misskes
Closd Models Onissions
Closed Models Overal Quality
0.60
0.000
3.0
030
224
27M
Figure 9: Performance of open and closed VLMSs on DOCENT, as measured by POSH. While open models are competitive when it comes to mistakes in their detailed descriptions, they lag behind in their omissions, covering less of DOCENT's reference descriptions than closed models.
26


Reviews:
[{"review_id": "uP9ufmIwO8", "reviewer": "Reviewer_6TR7", "summary": "This paper introduces a new metric for evaluating detailed image descriptions, termed POSH. POSH utilizes scene graphs as structured guidelines to direct LLMs in assessing fine-grained errors in image descriptions, such as compositional correctness. POSH offers a replicable and interpretable evaluation experience, outperforming existing metrics, including GPT4o. Additionally, the authors introduce a new dataset, DOCENT, which contains artwork paired with expert-written descriptions and model-generated descriptions. The dataset also includes annotations from experts, providing a challenging benchmark for evaluating the detailed description of images. Furthermore, the authors proof that POSH can be used as a reward function to achieve better performance than standard sft.", "questions": "* Why choose artworks as a benchmark? Instead of other more common or more representative domains\n* How to ensure repeatability, as there are probabilistic models used (qwen3)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and easy to follow.\n* Evaluating the image description is indeed a non-trivial task, and the proposed new metric for evaluating detailed descriptions is important for the field of image captioning.\n* I agree that a good metric for image-description should be grounded on fine-grained cues, localized on text spans.\n* The paper introduced the DOCENT dataset, which includes expert-written descriptions and annotations, and the quality is well controlled.", "weaknesses": "* POSH is reliance on a model to generate the scene graph introduces inaccuracies and errors, which could be a potential bottleneck for its effectiveness.\n* The use of scene graphs to evaluate image-text alignment has been discussed in previous papers like [1]; the authors need to clarify the uniqueness of POSH.\n* The proposed dataset covers artworks, but in practical applications, images of natural scenes are more common.\n\n[1] Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sZfXrgpfcy", "reviewer": "Reviewer_hCTY", "summary": "The authors propose PoSh, a reference-based metric for long-form image captioning. PoSh works by first constructing scene graphs for both the reference and generated captions. The metric then constructs templated questions for both the reference and generated captions' scene graphs and answers these questions with an LLM judge to determine the recall and precision, respectively, of the generated caption. As the metric generates numerous questions for both scene graphs, PoSh can yield both granular and coarse assessments of caption quality.\n\nThe authors validate alignment with human judgements on CapArena, a pre-existing benchmark, and DOCENT, a dataset containing artwork with expert-written captions that they introduce. PoSh outperforms baseline metrics for granular identification of mistakes and omissions and is either competitive with or outperforms existing metrics for coarser evaluations. The authors also validate their metric as a reward function for RL training, finding improvement over simple SFT.", "questions": "The metric's performance on the various evaluations already gives good signal regarding its quality. The paper would nonetheless be improved through an evaluation of the intermediate components of the metric itself. For instance, how accurate is the scene graph extraction (as measured via precision and recall against reference human-annotated scene graphs)? Alternatively, as this might be more feasible during the rebuttal period, how accurate are the LLM judge's answers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper has numerous strengths:\n- Firstly, the task of long-form image captioning evaluation is an important one as models continuously improve in capabilities. PoSh acts as an important contribution within this space by proposing a straightforward reference-based metric that converts the references and generations into scene graphs and using these to assess precision and recall. Particularly, the use of questions to assess both precision and recall lends the metric interpretability and granularity.\n- The method outperforms prior baselines for granular evaluations and is competitive with or better than other metrics for coarse evaluations.\n- The authors propose a new benchmark, DOCENT, with the novel domain of visual art. DOCENT is coupled with expert-written reference captions and human judgements for a range of vision-language models. This not only helps the evaluation of the metric but could also be used as a testbed for future metrics or for the generation quality of other vision-language models.\n- Additionally evaluating PoSh's performance as a reward model makes the evaluation of the metric very complete. I can imagine the granular nature of PoSh being also used to generate targeted natural language feedback for models.", "weaknesses": "The main weaknesses I can see are:\n- PoSh is going to be sensitive to the accuracy of the extracted scene graphs, where there could be errors either during the dependency parsing process or during coreference resolution. Figure 3 marking \"painting\" as a mistake acts as one example of this. \n- While I think PoSh could act as a strong reward model, it does presume access to detailed reference captions, which is expensive to curate on a large scale. PoSh being reference-based similarly restricts its use for evaluation to dedicated datasets for this purpose.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jvCIPdZDRD", "reviewer": "Reviewer_cNnP", "summary": "This paper presents POSH, a new metric for detailed image description. It computes scores grounded in fine-grained errors by adopting scene graphs to guide LLMs-as-Judge. POSH is interpretable and aligns better with human validation. Additionally, they propose DOCENT, a new dataset of artwork, references, and descriptions. They validate POSH on DOCENT and find it has better correlation with human judgments. By benchmarking open and closed-source models on DOCENT, they identify strengths and weaknesses of VLMs in understanding images.", "questions": "- Could you provide results with other VLMs other than Qwen2.5-VL?\n- What is the text and image distribution difference between DOCENT and CapArena? Is there any other caption benchmarks you could use to validate the effectiveness of POSH?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- They focus on an important aspect of VLM understandings. They focus on the detailed description, and the metric is interpretable. \n- They propose a benchmark with expert-written descriptions and 900 granular & coarse judgments from raters. The manual effort is massive.\n- They open-sourced the benchmark and metric, which will benefit the community. \n- They evaluate multiple open-source and closed-source models.", "weaknesses": "- The writing could be better. For example, in the table, they use POSH to denote the finetuned Qwen model with POSH reward, while POSH is a metric in the meantime. This is a bit confusing.\n- For the findings of POSH as a reward function, they only experiment with the Qwen2.5-VL-7B model. The findings may not be model-agnostic. \n- It is concerning that POSH works better on their proposed DOCENT benchmark but is adequate on other benchmarks like CapArena.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SQb53V358c", "reviewer": "Reviewer_XyHZ", "summary": "The paper proposes PoSh, a new metric for evaluating fine-grained image descriptions. PoSh extracts scene graphs from both reference and generated descriptions, preserving the object–attribute–relation structure, and uses these graphs as a structured scoring criterion to guide open-source LLMs in performing fine-grained judgments. This enables the identification of mistakes and omissions in generated text.\n\nThe authors construct a new benchmark, DOCENT, comprising 1,750 artworks from the National Gallery of Art, expert-written descriptions, outputs from VLMs, 300 fine-grained annotations, and 600 coarse-grained pairwise ratings provided by art history students. Experiments show that PoSh outperforms existing metrics on both DOCENT and CapArena. It can also serve effectively as a reward function in reinforcement learning.", "questions": "Do captioning models that incorporate structural priors (e.g., spatial or scene graph inputs) benefit disproportionately under PoSh?\nSome recent methods explicitly inject visual relational priors—such as object positions or scene graph structures—into the captioning process. \n1. Since PoSh itself uses scene graphs as evaluation rubrics, such models might receive inflated scores due to structural alignment between generation and evaluation, rather than superior semantic fidelity. Have the authors evaluated PoSh on outputs from graph-aware captioning systems (e.g., SG-LLaVA [1])?\n2. I also wonder whether the authors think that incorporating structural priors during generation might be more important than adding them during verification.\n\n[1] Jingyi Wang, Jianzhong Ju, Jian Luan, and Zhidong Deng. LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in Vision-Language Models.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. PoSh addresses a clear gap in evaluating detailed image descriptions. By using scene graphs as structured rubrics, PoSh enables error localization and produces human-aligned, interpretable scores.\n2. Built entirely on open-weight models and public tools, PoSh avoids reliance on proprietary APIs (e.g., GPT-4o), making it accessible and deployable for researchers with limited resources.\n3. The paper introduces DOCENT—a novel dataset of expert-written art descriptions paired with granular and coarse human judgments from domain-knowledgeable annotators. This enables evaluation for complex visual domains.", "weaknesses": "1. **Computational overhead may hinder scalability**: The pipeline, comprising scene graph extraction, multi-pass identifier generation, and LLM-based QA, is considerably more complex than standard metrics. The paper does not include ablation studies or timing analyses of individual components. Without such efficiency profiling, it remains unclear whether PoSh offers a favorable trade-off between evaluation quality and computational cost in large-scale settings.\n2. **Lack of validation as a data curation tool for MLLM training**: While the paper demonstrates PoSh’s effectiveness as a reinforcement learning reward signal, it does not explore its utility in filtering or ranking training data for MLLM fine-tuning. Comparing models trained on PoSh-filtered data versus those trained with other metrics would better establish PoSh’s practical value in the full model development lifecycle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)