Panoptic Captioning: An Equivalence Bridge for Image and Text
Authors: Kun-Yu Lin
Hongjun Wang
Weining Ren
Kai Han
Venue: NeurIPS 2025 poster ()
Keywords: Image Captioning
Comprehensive Image Captioning
Primary area: deep_learning

Abstract: This work introduces panoptic captioning, a novel task striving to seek the minimum text equivalent of images, which has broad potential applications. We take the first step towards panoptic captioning by formulating it as a task of generating a comprehensive textual description for an image, which encapsulates all entities, their respective locations and attributes, relationships among entities, as well as global image state. Through an extensive evaluation, our work reveals that state-of-the-art Multi-modal Large Language Models (MLLMs) have limited performance in solving panoptic captioning. To address this, we propose an effective data engine named PancapEngine to produce high-quality data and a novel method named PancapChain to improve panoptic captioning. Specifically, our PancapEngine first detects diverse categories of entities in images by an elaborate detection suite, and then generates required panoptic captions using entity-aware prompts.
Additionally, our PancapChain explicitly decouples the challenging panoptic captioning task into multiple stages and generates panoptic captions step by step. More importantly, we contribute a comprehensive metric named PancapScore and a human-curated test set for reliable model evaluation. Experiments show that our PancapChain-13B model can beat state-of-the-art open-source MLLMs like InternVL-2.5-78B and even surpass proprietary models like GPT-4o and Gemini-2.0-Pro, demonstrating the effectiveness of our data engine and method.
Project page: https://visual-ai.github.io/pancap/

Panoptic Captioning: An Equivalence Bridge for Image and Text
Kun-Yu Lin
Hongjun Wang Weining Ren
Kai Han"
Visual AI Lab, The University of Hong Kong
kunyulinOhku. weijingg .kuu
kaihanz Shhun hk
Abstract
This work introduces panoptic captioning, a novel task striving to seek the mini- text equivalent of images, which has broad potential applications. We take the first step towards panoptic captioning by formulating it as a task of gener- ating comprehensive textual description for image, which encapsulates all entities, their respective locations and attributes, relationships among entities, as well as global image state. Through an extensive evaluation, our work reveals that state-of-the-ar Multi-modal Large Language Models (MLLMs) have lim- ited performance in solving panoptic captioning. To address this. we propose an effective data engine named PancapEngine to produce high- high-quality data and novel method named PancapChain to improve panoptic captioning. Specifi- cally, our PancapEngine first detects diverse categories of entities in images by an elaborate detection suite, and then generates required panoptic captions using entity-aware prompts. Additionally, our PancapChain explicitly decouples the challenging panoptic captioning task into multiple stages and generates panoptic captions step by step. More importantly, named PancapScore and a human-curated test set for reliable model evaluation. Experiments show that our PancapChain- source MLLMs like Intern VL-2.5-78B and even surpass proprictary models like GPT-4o and Gemini-2.0-Prus demonstrating the effectiveness of our data engine and method. Project page: https: / s/vvivuulllll sei..iithnn github.......... io/iiii/aaa ho/pancpp/
contribute a comprehensive metric
- 13B model can beat state-of-the oof-the-an art open-
Introduction
Representing images by textual descriptions is fundamental topic in computer vision and natural language processing fields [1. 21. which benifits various applications, e.g., cross-modal retrieval [3, 4], multi-modal learning [5. 6, 7, 8, 9, 10], safe content generation [11. 12]. While prior works have explored various image caption formats, identifying the most effective format remains an open challenge. The most concise captions. which describe only primary entity categories, often sacrifice critical details like entity attributes. Conversely, highly detailed representations, such as paragraphs detailing all pixel-level semantics and their interrelations, are computationally burdensome due to their length. Inspired by these considerations, this work conceives finding the minimum text equivalent of an image, an ambitious yet challenging goal, which aims to develop concise textual description that comprehensively captures its essential semantic elements. Conceptually. achieving minimal text equivalence for images can be seen aligning images and text in the data space, while existing image-text alignment models like CLIP [13] perform this in the embedding space. Such text representations would maximize the utility of image information for learning and downstream applications.
This work introduces the task of panoptic captioning, which strives to seek the minimum text equivalent of images. Our work serves the initial effort towards this challenging task. To make the
*Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

with the light peing The image shows a huskyy dog sitting on the ground curdoors, I1 is sunny day. bright, casting soft shodows, and the
Panoptic Caption:
Val-Atir
to duning
the daytime, In the foreground, deg, positioned at (115, 534, 1288, 2039), wears t-shirt and bock. The
Val-Rel
Val-Loc
piece of fabric draped
mostly this headband locoted (850,
groy
black
morkings,
headband ground is looking aiihhtly
442, 630), Its cars one perised up,
right
wearing
St
t-shirt with blue steewes, The 1-shirt within the bounding box [410, 964, 1169, 1648] The shirt has writing on the front of it, which composed of two lines of texts. The text
Test-Tag
s2 12 Taakn Tag
CONFEDERATIEE TIONS CUP appears on the tishirt in curved line above the text "RUSSTA 2017 longer fort. The writing's bounding box [674. 1067. 1100, 1248]. A red fabric, positioned (357, 1519, 1071. 1912] possibly pants
25.2
To the right the dog, part persan's visible, weoring oronge ponts,
bounding
[1326,
1495,
Test-Loc
Test-Rel
brown soncal. Only background. part of a blue con can be seen with bit of dork space under the vohicle. The bounding box of the car is [0, 1500, 9411. The ground, positioned at (0, 720, 1500, 2254], is groy aspholt surface, Towerds the bo toot-n-i of the image, there are white zebro markings pointed on the asphalt. The markings"
1694). person's
weaning yellow sock
2/
the lower leg. from just below knoe down, visible
Test-Attr
IntemVV1-2.5 1...v/1--2.5-000
Gemini 2.0 Pro
bounding box is (765, 2019, 1497. 2254).
1lama- 3.2-908
PancapChaiee 13BB
Reconstructed Images from bifferent Types of Captions
Coption:
BLIP-2's Short
Share6PT4V's Detailed Caption: In this dog sitting the ground front of blue boat.
PancopChair''s Panoptic Caption (Ours): This image shows dog +boxi[]200, 300, 1290, 2000]]//bol /oooo sitting on the ground front blue...
dog weaning olympic 1-shirt.
The dog's a
Top Left: An example to demonstrate our proposed panoptic captioning task, which is formulated as generating a comprehensive textual description encapsulating all entities, their respective locations and attributes, relationships among entities, as well as global image state for a given image. Top Right: We report several models' performance on four distinct dimensions on the validation and test sets of our SA-Pancap benchmark. The figure shows that three state-of-the- art Multi-modal Large Language Models (MLLMs) struggle with panoptic captioning, while our proposed PancapChain performs generally better with a significantly smaller model size. Bottom: Image "reconstruction" " by the toxt-to-image model PixArt-E [14] with different types of captions. Best viewed in color.
Figure 1:
problem tractable, we formulate panoptic captioning as the task of generating a comprehensive textual description for an image, which captures all entity instances, their respective locations and attributes, relationships among entity instances, as well as global image state (see Figure (top left)). This formulation serves as a reasonable approximation to our conceptual "minimum text equivalence", which captures basic semantic elements (e.g., center dog) for comprehensiveness while excluding less critical subtle details (e.g., tiny particles on the ground) for conciseness. In contrast to prevailing captioning works that vaguely specify locations by pure words [15, 1, 2, 4, 16, 17, 5]. e.g., BLIP-2's brief captions and ShareGPT4V's detailed captions, our panoptic captioning stands out for its text comprehensiveness. By accurately localizing entity instances using bounding boxes, it offers an effective and efficient way to describe position and occupied region of an entity instance with only several coordinate numbers. Figure (bottom) shows our panoptic captioner performs better image "reconstruction" from captions due better comprehensiveness.
Due to the fundamental formulation differences from existing captioning works, their metrics cannot effectively evaluate model performance in panoptic captioning. To this end, we propose new metric named PancapScore. which groups semantic content into distinct dimensions based on task formulation and then conducts evaluation each dimension. PancapScore first evaluates semantic tagging and instance localization by entity instance matching, and then assesses attribute, relation and global state in flexible question-answering manner. Our experiments demonstrate that PancapScore aligns closely with human judgement. Based on PancapScore, we conduct a thorough evaluation on state-of th-the- Multi modal Large Language Models (MLLMs). As shown in Figure 1 (top right), existing MLLMS have limited performance in panoptic captioning, revealing its challenging nature.
To address this new and challenging task, we propose an effective data engine named PancapEngine to produce high-quality data in a detect-then-caption manner. Specifically, we first detect diverse categories of entities in images, by associating class-agnostic detection with image tagging. We then employ state-of-the-ang MLLMs to generate comprehensive panoptic captions using entity-aware prompts, while ensuring data quality by caption consistency across MLLMs. Based on PancapEngine, we contribute a new SA-Pancap benchmark composed of high-quality auto auto-generated data for training and validation, and additionally provide human-curate test set for reliable evaluation.
Furthermore, we propose a novel method named PancapChain to improve panoptic captioning. The key idea is to decouple the challenging panoptic captioning task into multiple stages and train the model to generate panoptic captions step by step. PancapChain first localizes entity instances, then assigns semantic tags to instances, and finally generates panoptic captions. Surprisingly, our

---

PancapChain- 13B model beats bate-offthe art large open- spen-source MLLMs like Intern VL-2.5V--. and even surpasses proprietary models like GPT-4o and Deminni-2.0-Pro Additionally, our model can facilitate downstream image-text retrieval tasks by transforming images into captions and performing text retrieval. Table shows that our model achieves superior performance on the challenging DOCCI dataset, which outperforms a state-of-the-ar image-text alignment model ALIGN without specialized training data and module designs, and beats the state-of-the-an detailed captioner ShareGPT4V. These experiments in panoptic captioning and downstream image-text retrieval demonstrates the effectiveness and application value of task and model.
Table I: Image-text retrieval re-
sults DOCCI [4] in Recall @ 1.
Models
Model Type
R@1 1
Our contributions are summarized as follows: First, we introduce the novel panoptic captioning task, which strives to seek the min-
ALIGN [18]
Image-Tent Text
$9.9
BLIP [15]
Text-Text
47.3
imum text equívalent of an image-an an ambitious yet challenging goal. We formulate it as the task of generating a comprehensive textual description composed of five distinct dimensions, and con- tribute comprehensive PancapScore metric for reliable evaluation. Second, we propose an effective data engine named PancapEngine to produce high-quality data. We also contribute the SA-Pancap benchmark for model training and evaluation, which includes a high-quality validation set and human-curaan test set for reliable evaluation. Third, we propose a simple yet effective method named PancapChain to improve panoptic captioning, which decouples the challenging panoptic captioning task into multiple subtasks. Extensive experiments demonstrate the effectiveness and value of our task and model.
ShareGPT4V [5]
Text-Text
59.6
PancapChain
Text-Text
61.9
2 Related Work
Image Captioning, Image captioning is a fundamental topic in computer vision and nature language processing fields [1, 2]. It aims describe the visual content of an image using meaningful and syntactically correct sentences. In early years, many methods have been proposed to address image captioning, mainly based on RNNs and Transformers [19, 20, 21, 22, 23, 24, 25, 26, 27, 28]. Traditional image captioning adopts evaluation metrics like BLEU [29], METEOR [30] and CIDEr [31], which leverage N-gram and lexical similarity with numan-annotarn
captions. Above early works focus on short captions lacking details. Additionally, scene-graph graph- based captioning methods [32] first generate scene graphs and then produce captions. However, these methods are restricted to fixed, predefined categories of objects and relationships, and their performance is limited by the subsequent caption integration process and the need for additional models. Recently, owing to the development of vision-languee
pre-training, Multi modal Large Language Models (MLLMs) have been equipped with the capabilities of generating detailed image captions [15, 33, 34, 35, 17]. These detailed captioning works also promote the development of MILMs, e.g. using detailed captions pre-training boosts performance on downstream tasks [5, 6]. Due to the unstructured nature and rich semantic complexity of detailed captions, evaluating model performance in detailed captioning presents significant challenges. Recent works have developed various types of metrics to evaluate long captions [36, 37, 38, 39, 40, 41, 42, 43, 44].
Previous captioning works vaguely specify entity locations by pure words. Unlike these works, our panoptic captioning introduces a more comprehensive and economic caption format, which accurately localizes entity instances using bounding box coordinates. This enables better descriptions for the positions and occupied regions of entity instances, and helps distinguishing instances with similar attributes. Accordingly, we propose a new PancapScore metric for comprehensive evaluation. Another related task is dense captioning [22, 45], which generates short captions for individual regions in an image. Typically, this task focuses on a limited set of entity categories and does not account for the relationships between entities. Some recent Grounded MLLM works have explored the joint task of image captioning and visual grounding [46, 47, 48]. However, they rely on additional specialized modules to achieve localization capabilities, and they usually generate brief captions lacking details for a whole image (e.g., GLaMM [46]). In contrast, our work integrates localization capabilities into detailed captioning through pure textual descriptions using unified LLM, taking an initial step to explore the minimum text equivalent of images.
Vision- Language Vision-Language Models. Recently, significant advancements have been made in Vision-Language Models (VI.Ms.. A typical type of VL.Ms is based on image-text matching [13, 18, 49, 50, 51, 52, 53, 54], which achieves remarkable zero-shot image understanding performance and initiates the era of
3

---

open-world understanding [55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 691. Another type of isMs called Multi- Large Language Models (MLLMs) [34, 70, 71, 72, 73, 74, 75, 70, 76], which is motivated by the remarkable success of Large Language Models (I.LMM) [77, 78, 79, 80] in instruction following [81], in-context learning [82] and reasoning [83]. MILLMs propose to integrate the power of LLMs with multi-modal perception and comprehension, which enables multifunctional visual understanding and shows remarkable performance in various tasks. In addition, recent Grounded MLLMs explore integrating location- -waree understanding abilities into MLLMs, enabling region-aware downstream tasks [84, 85, 86, 87, 46, 88, 89, 90, 91, 47, 48]. Although significant efforts have been devoted to advance MLLMs, work reveals that state-of-the-ar MLLMs exhibit limited performance in panoptic captioning, which is fundamental for image and multi-modal understanding.
3 Panoptic Captioning
3.1 Task Definition
In this work, we formulate panoptic captioning as the task of generating comprehensive textual description for an given image, which encapsulates all entity instances, their respective locations and attributes, relationships among instances, well global image state. Specifically, we group the semantic content in panoptic captions into five dimensions, which are detailed as follows:
Semantic tag refers to the category label assigned to each entity instance in an image. Panoptic captioning requires identifying all entity instances and assigning category label to each instance. Following Kirillov et al. [92], we define "entities" as both countable objects (things) such people and animals, and amorphous regions (stuff) such as grass, sky, and road.
Location refers to the spatial positions of entity instances, which represented in terms of bounding boxes. Specifically, following previous detection works [93], the bounding box of an entity instance is denoted by (x1, y1, F2, 92).. where (I1, //) and (x-2, y2) denote the top-left and bottom-right corners of the box, respectively. By introducing bounding boxes, panoptic captions can more accurately describe the locations and occupied regions of entity instances, which also helps distinguishing entity instances with similar attributes more easily.
Attribute refers to characteristics or properties that describe an entity instance's appearance, state or quality. The attribute dimension encompasses wide range of semantic content types, e.g., color, shape, material, texture, type, text rendering. Describing attributes can provide a detailed understand- ing for an entity instance, enabling accurate entity identification and image analysis [94, 41, 433. Relation refers to connections or interactions between different entity instances within an image. The relation dimension encompasses wide range of semantic content types, such as position relation (e.g., A is behind B). whrle relation (e.g., A is part of B) and action relation (e.g., kicks B). Describing the relations between entities is crucial for understanding the image's 's structure, context, and dynamics [94, 43].
Global image state refers to the overall characteristics of an image that provide holistic understand- ing of its content, without foeusing on specific entity instances within the image [94]. For example, global image state includes lighting conditions and color palette.
Figure (top left) demonstrates an example for the task definition. Overall, by considering the above five distinct dimensions, panoptic captioning leads to a comprehensive textual representation by capturing all basic semantic elements. It should be noted that our proposed panoptic captioning differs from conventional captioning works [15, 5, 4, 16, 17] mainly in text comprehensiveness. Instead of vaguely specifying entity locations by pure words, panoptic captioning requires models accurately localize entity instances using bounding boxes, which enables better descriptions for positions and occupied regions of instances and helps distinguishing instances with similar attributes more easily.
3.2
The PancapScore Metric
Prior works in detailed captioning [43, 41] have demonstrated that evaluating the quality of detailed captions presents significant challenges due to their free-form nature and rich content. Evaluating models in panoptic captioning becomes even more complex, as it requires an additional assessment of bounding box accuracy. To address this challenge, we propose a new metric named PancapScore for comprehensive and reliable evaluation. PancapScore systematically categorizes the content in panoptic caption into five distinct dimensions and evaluate model performance on each dimension separately, faithfully aligning with the task objective.

---

Reference Caption
Semantic Content Extraction
Entity Instance Matching
The image shoad taaity cunceers Tr is sunyy day. with the light being ewon stadows and...
dog [125, 334, 1268, 2039]
Attribute 1 mestly whrte
I. dog
1. leg
2. leg: (1376, 740, 1460, 1694]
2. lig
2. dog
sitting
car
B. R-shirt
720, 1500 2794]
Subal:
graadd
vehicle
Instance-aware QA
imoge festures dag drassed aitting on poved surface. everall enoramment beight, clean lighting condiiine...
1. loy (980, 1355, 1485, 2213]
Aftribute 1D 1 weer yollow
2. dog (426, 951, 1066, 3607]
Attribute Is Ib nearing ye.
Relation to left ID
Selation:: sohvnd 27
(.. vehicle, (0, 2500
Global The ovend entronment
Generated Caption
Gliboll To the overall ovirern...
An overview of our proposed PancapScore metric. PancapScore first extracts semantic content from captions, and then evaluates model performance by entity instance matching and instance-aware question answering (QA).
Figure 2:
Specifically, given an image, PancapScore evaluates the quality of generated panoptic caption using the ground-truth caption as the reference. PancapScore first extracts all semantic content from captions and groups them into five dimensions. Based on extracted semantic content, PancapScore evaluates semantic tagging and instance localization based on entity instance matching. Then, PancapScore evaluates attribute, relation and global state in a question-answe manner, and finally obtains the overall score considering all five dimensions. An overview of our proposed PancapScore metric is given in Figure 2.
Semantic Content Extraction. First of all, we propose to extract all the semantic content from generated caption and group these content into five dimensions, namely semantic tag, location, attribute, relation and global state. The content extraction is implemented using state-of-the-art Large Language Models (LLMs), as they have strong understanding capabilities on textual descriptions and instructions. To facilitate the understanding of LLMs, we carefully design prompts with in-context text examples included.
Figure includes an example of the extracted semantic content. Specifically, the extracted semantic content is organized item by item, and each item is basic unit of semantic information. The leading items specify the semantic tags and locations of entity instances. Each entity instance is associated with unique semantic-tag item with unique instance ID. The following items describe attribute and relation information. Each of these items is associated with an instance ID, indicating that describes the attribute or relation of the corresponding instance. The last items describe the global state, and these items are not tied to specific instances. During evaluation, we will separately extract semantic content from the generated and reference captions.
Entity Instance Matching, After extracting semantic content, we conduct an instance matching between the generated and reference captions to evaluate semantic tagging and instance localiza- tion. Formally, the ground-truth and predicted entity instance sets are denoted by {(t,, 6,)) and ((,,,,,)) b,)) (4, b,)/(C,, b,) denote the semantic tag and bounding box of the i-th ground- cruth/predicted instance. n and m denote the total number of instances. Accordingly, the entity instance matching is formulated as an optimal one-to-one matching problem as follows:
T O arg maxo..; En E (p (p- + iou(b, b,)) Or.51
where O is an assignment matrix and Oig 10 indicates if (4:, b.) and (bz,b;) are matched (i.e., Oz, = 1 indicates that the i-th ground-truth instance matches the j-th predicted instance). St, denotes the similarity between f; and ły. which considers synonyms and embedding similarity. iou(bs, b,) denotes the box IoU, and μ 10 is weight coefficient to raise the priority of tagging.
Based on the entity instance matching, measure the model performance in semantic tagging and instance localization. First, we consider two matched instances to be semantically consistent if their tags share synonymous nouns or have similar text embeddings, i.e., Sty 2 8,. By considering all entity instances, we compute precision and recall in semantic tagging. Precision measures the proportion of correctly predicted tags among all predictions, while recall measures the proportion of correctly identified ground-trutt instances. Based on precision and recall, we compute the F-score for an overall measurement for semantic tagging. Besides, we consider two cemantic-consistrn instances to be neation-consistenn if iou(le b,) > 6,. Following a similar way in semantic tagging, we obtain F-score for instance localization by localization consistency. Please see Appendix for more details.
5

---

Instance-aware Question Answering, Based on entity instance matching, we evaluate models' performance in attribute, relation and global state in a flexible question -anseerig manner. First, we generate questions based on the semantic content of the generated caption, where each question verifies whether an atrribute/relationnlloaa state item is described in the reference caption. For each atrribute/relation item, the associated instance IID in the generated caption is mapped to their corresponding instance ID in the reference caption. For example, the generated caption contains an item "ID 2 is red" and the "ID 2" corresponds to the "ID 3" in the reference caption, we generate question like "Is ID 3 red?". To enable robust evaluation, we generate a pair of questions for each semantic item: one with a "Yes" answer and the other with "No" answer.
We employ a sfate-of-the-ar LLM to answer questions according to the reference caption. In this process, we instruct the employed LLM with a carefully designed prompt. where in-context text examples are included. If the LLM outputs the same answers as the preset ones, we consider the cor- responding semantic item to be correct prediction. We measure precision in atrributerrelaiionnlloo state by computing how many questions are correctly answered. Additionally, we measure recall by generating questions from the reference caption and evaluate the LLM's answers based on the generated caption. Based on precision and recall, we obtain F-scores for an overall measurement in attribute, relation and global state.
Overall, our PancapScore metric includes five Fi-scores from dimensions of tagging, localization, attribute, relation and global state. The five scores are denoted by St, $1, Sa, S, and sy. respectively. Based on these scores, the overall score of our metric is formulated as: s = 8 + St + S, + S, + Sg Since an image usually has much fewer global state items compared with other dimensions (i.e., usually one or two items), we introduce the coefficient Xg 0.1 to downweight the global state's score. Experiments in Appendix show that PancapScore aligns closely with human judgement.
4 Data Engine and Benchmark
4.1 PancapEngine
To address this new and challenging task, our work proposes an effective automated data engine named PancapEngine to produce high-quality data. Our PancapEngine first detects diverse categories of entities in images using an elaborate entity detection suite. We then employ state-of-the-ar MLLMs to generate comprehensive panoptic captions using aatity-aware prompts, ensuring the data quality by caption consistency across diffcrent MLLMs.
Entity Detection Suite. Existing detection or segmentation datasets are often limited to small number of predefined categories thus directly using these datasets will limit the entity diversity of panoptic captioning data. To improve entity diversity, we propose to associate class-agnostic detection with image tagging for detecting diverse categories of entities in a given image. Specifically, we first detect entity instances using& ofate-of-the-art class-agnostic detector OLN [95], and the resulting set of regions is denoted by R. We then assign semantic tags to regions by a state-of-the-art image tagging model RAM [96]. For each region in R. we crop the region from the image and feed it into RAM to obtain its semantic tag. RAM can recognize 6,400+ common entity categories, which is much more diverse than cxisting detection datasets. In addition, we integrate two specialized class- aware detectors, namely Grounding-DDin DINO [99] and OW-DETR [100], to identify instances missed by OLN. We aggregate all entity categories from OLN's s detected regions and utilize this aggregated category set as input prompts for Grounding- DINO and OW-DETR to enable class-aware detection. The resulting region set from class- detectors is denoted by R'. We then merge the two sets R and R', and remove redundant regions based on IoU, In this case, we will add a region from R' to R, if this region has low IoUs with all the regions in R. We do not use non-maximum suppression to remove redundant proposals as different detectors produce confidence scores in varying ranges. With comprehensive set of detected entity instances, we proceed produce detailed panoptic captions.
aware Caption Generation. Based on detected entity instances in images, we construct entity-aware prompts and instruct state-of-the-ars MLLMs generate panoptic captions. An entity- aware prompt includes all detected entity instances in the query image, associated with semantic tags and locations. In addition, the prompt explicitly specifies attribute, relation and global state types to help MLLMS discover more semantic content. In-context examples are also included in the prompt for better instruction following. We employ Gemini-Exp: 1121 and Qwen2-VL-72B
For example, COCO [93, 97] includes 80 objects and 91 stuff, and Object365 [98] includes 365 objects.

---

to generate required captions due to their strong image understanding and instruction-following capabilities. Specifically, we first employ Gemini- Expini-Expp11211 to generale required captions using entity-aware prompts. Similarly, we employ Qwen2-V1.-722 to generate required captions and conduct quality verification by caption consistency. We drop a caption generated by Gemini-Exp- 1121, if it has low consistency with the corresponding ewen-generated caption. We adopt our PancapScore metric to measure caption consistency, without considering the location dimension. This is because we empirically find that Qwen2-VL-72B VL-72B has much weaker localization capabilities than Gemini-Exp- 121 when entity-aware prompts are given. Overall, by entity- generation and quality verification, our PancapEngine can generate high-quality panoptic captions.
4.2 The Proposed SA-Pancap Benchmark
Based on our PancapEngine, we contribute new SA-Pancap benchmark for the panoptic captioning task. We select SA-IB [101] as the data source due to its high image quality and data diversity. Overall, our SA-Pancap benchmark consists of 9,000 training and 500 validation images paired with auto-generated panoptic captions, and 130 test images paired with human-curater panoptic captions.
Our validation and test sets consist of diverse images, paired with high-quality panoptic captions. Specifically, based on PancapScore, we select the images paired with the most high-quality panoptic captions to construct the validation set. Additionally, we ask human labors to refine model-generated panoptic captions of the test set, following rigorous curation process. To ensure the data diversity of the validation and test sets, we leverage DINOv2 [102] to select distinctive images. Specifically, we enforce a DINOv2 feature similarity threshold of 0.2 between any two images within these sets, thereby preserving high degree of visual vari- ability. Additionally, we enforce a DINOv2 fea- ture similarity threshold of 0.5 between training images and validation/test images.
Table 2:
Compared with previous benchmarks, our SA- SA-Pancap provides more comprehensive cap- tions, associated with diverse entity instances and accurate location annotations.
Benchmarks
Location Instance Category
Sample
Token
DCI [103]
7.8K
148.0
In summary, SA-Pancap consists of high-quality panoptic captions, which comprehensively covers diverse categories of entities and rich semantic content in location, attribute, relation and global state dimensions. Table 2 shows comparison between our SA SA-Paneap with previous captioning benchmarks. Compared with detailed captioning benchmarks, our SA-Pancap has more detailed captions with more tokens and associates entity instances in captions with location annotations. Compared with GCG [46] involving short captions, AA-Pancap provides much more comprehensive captions with diverse categories entities and more instances per image.
DOCCI [4]
14.6K
135.7
IIW [16]
217.2
SG4V [5]
X
1.2M
192.0
DenFu [6]
1.0M
254.7
GCG [46]
2.9
1329
56.9K
27.2
SA SA-Pancap
6.9
2429
9.6K
345.5
5
ChaiapChain
To improve panoptic captioning, we propose novel method named PancapChain. Our key idea to decouple the challenging panoptic captioning task into multiple stages and train the model to generate panoptic captions step by step, as an image contains rich semantic elements. This is inspired by cognitive science works [104, 105], which has demonstrated that humans struggle to perceive and comprehend all elements within complex scene simultaneously. Accordingly, given an image Q", our PancapChain proposes to generate panoptic caption Á by four stages, namely, entity instance localization, semantic tag assignment, extra instance discovery, panoptic caption generation, denoted as S (Lo,, Tag. Disc. Cap) We show an overview of PancapChain in Figure 3 and detail it as follows.
Entity Instance Localization (S In the first stage, we propose localize entity instances, as entity instances are the foundation for describing images. To this end, for the image Q", we extract the bounding boxes of instances from the ground-truth caption A, and construct an image-text pair {Q", A ") for training. A J. is the localization text consisting of bounding boxes of all instances, which are concatenated by commas. Following ASMv2 [85], each bounding box represented by the text in the format of "<box> [[x) y1, x2, 12] </box>", where (œ1, y1) and (I2, y2) denote the lept and bottom-right corners of the box. All bounding boxes are normalized to integer values within [0, 1000), and images are resized to a fixed size.
7

---

Entity Instance L Localization
Semantic: Tag siggmment
Panoptic Caption Generation
Prompt: Localize all entrries thi imgg
Specify the semontic estities bosed on their bau brunding hoves
Prompt:
Prempt:
Prez hypar-datulled descrighion for tiis irage. including all ertifies, their locations, aftributes, and reletionchips. of as global image state
Tagging
+bae 334,1286 2/393/bees choso (410 904, 1169 1648 y/bss.
SegeboooiD1o 334, 334,288 2039)//eer, bathifebeee[4400 966 1169 1648)e/boso, Febrte (337 1019 2071 /o
chesn (337 1519 1071, (0te whesoo
In the foreground dog is posthioned of 2039) [115, headband
rbaxx]]5326,44 1694//o 2
bypbse|11326 740,1495, yy54
Blend
Localization
headhandd loctte
642, 990, 6600) Tt is wear white tithir with blue sleeves The I shhrt within the bounding box (150,
Extra Instance Discovery
Pronpt:
Specifyy the wicsing entitigs hiss and their this image based on
shirt
Discovery
(674, 5067. 1100 1245). it reads RRUSSAA 2017*. red tuoni positiines at (1 1819 1011 1010) dropedt around the doy's lowerback. the right of dag. port person's log visibee...
headhands baso (1850 442 990 A30)///o
tast <baxs[1674, (067 1000, 4245///00o
corviosel[O (,10000443 941)///o
Ieput Tmago
Figure 3: An overview of our proposed PancapChain method. PancapChain explicitly decouples the challenging panoptic captioning task into four stages, namely entity instance localization, semantic tag assignment, extra instance discovery and panoptic caption generation.
Semantic Tag Assignment (Stag). In the second stage, based on the localization text, we propose to assign semantic tags to the localized entity instances. To this end, we extract the semantic tags of instances from the ground truun caption, with each associated with one bounding box, and then construct an image-text tuple {Q", A4, , for training. A' is the instance text consisting of seman- tic tags and bounding boxes of all instances, which are concatenated by commas. Specifically, each entity instance is represented by the text in the format of "tag <box> [[x1, T2, y2]] </bo>>" where "tag" denotes the ground-truth semantic tag. For example, "dog <box> [[100, 200, 500, 600]] </box>" means there is a dog located within the box [100, 200, 500, 600]. In this stage, A 4 is introduced in the textual prompt to instruct model training. During inference, the predicted location text A L. is included in the prompt.
Extra Instance Discovery (S Since an image contains numerous entity instances, it is not trivial to identify all instances at once. Therefore, introduce additional stage detect instances that are missed in the first two stages. Specifically, for the image Q", we construct an image-text tuple [Q"AA, A1} for training, where A is included in the textual prompt and A is used as the ground-truth for supervision. We randomly split the ground-truth instance set into two parts, and construct two instance text namely Af and A.g. The text Af contains boxes and tags of those instances that have been discovered, while A indicates the other instances to be discovered. In our design, A contains fewer instances than A to simulate the discovery of previously missed instances. During inference, we include the predicted instance text À in the prompt and then predict the extra instance text A
Panoptic Caption Generation (Scap). Finally in the fourth stage, we generate panoptic captions based identified entity instances in early stages. Formally, construct an image-text tuple {Q", (Q",A, A} for training, where A is the ground-truth panoptic caption of the image. The bounding boxes in A will be transformed into the same format as those in entity instance localization. The instance text A' is included in the textual prompt for training. During inference, we should first aggregate the initial instance text A' and extra instance text Å F and include the aggregated instance text in the prompt to predict caption Å.
Overall, the training loss of our PancapChain is formulated as: L(Ā", A") + L(À', A') + E AA)+ L(AA,A))++ L(A, A). where L(---) denotes the standard auto-regressive loss following LLaVA [34]. Different prompts are used in these four stages to instruct model training. During inference, our model generates captions stage by stage, according to the guidance of prompts.
6 Experiments
Implementation Details. Our model adopts the general LLaVA architecture [34], and it is initialized using the pre-trained ASMv22113B [85] checkpoint due to its good grounding capabilities. We finetune our model on the training set of our SA-Pancap for two epochs using LoRA. For PancapScore, we use Qwen2.5-144 14B [106] as the L.MM for semantic content extraction and question answering. The thresholds are set as & 0.5 and 6 0.5. Please see Appendix for more details.
8

---

Table 3: Comparison with state-of-the-art MLLMs on the validation and test sets of SA-Pancapp Performance are measured by PancapScore, and we report the scores in tagging (Tag), location (Loe), attribute (Att), relation (Rel), global state (Glo), and the overall score (All). Bold/underlined indicates the best/second-bert We prioritize the overall score as the primary metric in model comparison.
Validation
Test
Models
Scale
Tag
Loc
Att
Rel
Glo
All
Tag
Loc
Rel
Glo
All
Molmo [108]
72B
52.06
10.03
36.88
25.90
76.78
132.53
50.92
14.00
38.10
19.96
68.49
130.55
LLaVA-OV [109)
72B
54,20
13.79
38.94
27,80
85.52
143.28
53.62
15.16
41.52
25.63
82.39
144.17
Qwen2-VL [107]
72B
49.85
12.92
37.83
24.71
86.30
133.96
48.19
12.90
38.48
20.44
84.13
128.42
Qwen2.5-VL [86]
72B
54.08
19.70
40.00
27.24
85.34
149.54
54.42
25.11
42.33
26.32
87.12
156. 89
NVLM [110]
72B
54.69
10.78
42.49
30.40
86.21
146.97
57.79
11.53
46.48
29.48
78.60
153.14
Intem (111)
78B
54.68
15.05
41.81
27.41
88.37
147.79
55,90
18.26
43.63
28.72
81.46
154.66
Llama-3.2 [35]
90B
52.87
20.73
39.94
27.09
83,40
148.98
51.64
21.88
40.55
25.33
79.55
147.35
GPT-4o [78]
Proprictury
50.89
10.12
40.54
25.40
$8.85
135.83
53.51
14:55
43.86
27.38
87.08
148.01
Gemini-2,0-Pro [112]
Proprictary
53,79
16,66
43.14
28.52
86.50
150.75
53.89
21.59
45.62
27.99
87,91
157.88
LLaVA-1.5 [74)
13B-Tuned
54.92
27.76
41.27
28.69
81.94
161.84
54.33
30.57
41.81
30.62
75.73
164.92
ShareGPT4V [5]
13B Tuned
55.02
23.81
40.53
29.13
82.16
156.70
52,94
25.56
39.56
25.11
80.36
151.21
PancapChain (Ours)
13B-Tuned
57.56
30.34
44.78
34.61
84.59
175.75
56.45
31.76
44.46
32.54
173.19
Comparison with State-of-the-Arss. Table 3 summarizes the comparison results between our proposed PancapChain and current ofate-of-the-art MLLMs on SA- SA-Pancap. In this experiment, we carefully design prompts for these MLLMs to unleash their panoptic captioning power, where in-context examples are introduced for better instruct-followii
From the table, find that current MLLMs struggle with panoptic captioning, except the global state dimension. This is attributed the challenging nature of the task, which requires models to comprehensively capture semantic content on five dimensions in images. We find that grounding capabilities are important for panoptic captioning, as Qwen2.5-VL, [86] significantly outperforms Qwen2-VL. [107]. By finetuning our training set, our PancapChain- 13B model significantly outperforms all state-of-ther ahe-art MLLMs in terms of the overall PancapScore metrie. Also, except the global state, our PancapChain 13B usually achieves the best or recond-best on each individual dimension. More importantly, our model is much smaller than state-of-the-arr MLLMs, which demonstrates the effectiveness of our proposed data engine and model design. By directly finetuning on the training set, LLaVA-1155 can also obtain good results, which demonstrates the high quality of our training data. Furthermore, our PancapChain still outperforms the tuned LLa model, which demonstrates the effectiveness of our design.
Ablation Study. We use a one-stage model as our baseline, which is directly finetuned the training set of SA-Pancap without specific designs. To improve panoptic captioning, we first introduce an extra instance discovery stage based on the baseline. This model variant first predicts semantic tags and locations of instances for an input image and then generates panoptic captions based on predicted tags and locations. Table 4 shows introducing this instance discovery stage improves the performance on attribute and relation dimensions, since we decouple the challenging task into two relatively easier subtasks. Next, we de- couple the instance discovery stage into entity instance localization and semantic tag assignment stages, as introduced in Sec. 5. As shown by "w/ S Tag)" ", this variant obtains notable improvement in se- mantic tagging. As shown by "Full (w/ S|Loe, Tag, Disc))", by further introducing the proposed extra instance discovery stage, our model obtains notable overall improvement.
Table Ablation study on the validation set of SA- Pancap. SLoc, Tag, Disc refer to our entity instance local- ization, semantic tag assignment and extra instance discovery stages, respectively. The caption generation stage Scap is required in all cases.
Models
Tagging Location Attribute
Relation
Baseline
56.38
29.30
43.06
31.64
w/ Instance Discovery
56.47
29.61
43.71
32.62
w/ ((Lc, Tagl
57.04
29.83
43.76
33.69
Full (w/ S [Lo, Tug. Disc))
57.56
30.34
44.78
34.61
This is because that model accurately identifies more entity instances and accordingly boosts the prediction on attribute and relation dimensions. In summary, our full model improves the baseline by 6.5+% in terms of the overall score.
Image Reconstruction. To qualitatively demonstrate the effectiveness of our model, we conduct an image reconstruction experiment by associating captioners with -nntte generation models.

---

Original Image
PancapChain
Qwen2.5-VL
ShareGPT4Y
BLIP-2
Figure 4: Image reconstruction using PixArt-E [14]. Compared with baseline models, our Pan- capChain can better capture image details, and thus lead to better image reconstruction.
This experiment serves as a proxy for evaluating the completeness of image descriptions, i.e., if caption captures all essential visual elements, a text-to-image model would be able to reconstruct an image similar to the original one. Specifically, we generate a caption for an input image using a captioner, and then adopt a text-to-image generation model PixArt-E [14] to generate a new image. As shown in Figure 4, PixArt-E associated with our PancapChain model can generate more similar images to the original images than baseline models, which demonstrates the effectiveness of our model in comprchensively capturing image details. Please refer to the Appendix for more implementation details, results and discussions.
For more experiment results, e.g., image- text retrieval and ablation study, please refer to the Appendix.
7 Conclusion and Discussion
This work introduces panoptic captioning, a novel task striving to seck the minimum text equivalent of image- ambitious yet challenging goal. We take the first step towards panoptic captioning by formulating it as a task of generating a comprehensive textual description composed of five dimensions. To study this task, we proposed comprehensive metric for reliable evaluation, designed an effective data engine to produce high-quality data, and established a benchmark for model training and evaluation. To address this task, proposed decoupled learning method to generate captions step by step. Extensive experiments demonstrated the effectiveness and value of our task and model.
Our work reals that, despite remarkable progress in generating detailed textual descriptions, existing MLLMs still struggle with panoptic captioning. demonstrating their limitations in bridging image and text modalities. Also, evaluation provides insights into the performance gap between open-source and proprietary models. By finetuning on high-quality data, our PancapChain- 13B model beats state- of-the-art art MILMs, e.g., Intern VL-2.5-78B and Bemini-2.0-Proo highlighting the potential of our task and model in bridging image and text modalities. Additionally, when paired with a text retriever, our model achieves superior performance in downstream image-text retrieval, demonstrating the practical utility of panoptic captioners. We believe that, developing more powerful panoptic captioners benifits various downstream applications learning tasks, e.g., multi-modal understanding.
Despite our efforts, panoptic captioning still faces numerous unresolved challenges, and our task formulation remains approximation, not yet fully achieving ultimate conceptual goal of "minimum text equivalence". Nevertheless, our work lays a solid foundation for future development, and will act catalyst to accelerate the progress of developing textual representations for images.
Acknowledgments. This work is supported by the Hong Kong Research Grants Council General Research Fund (Grant No.: 17211024). The authors would like to thank Jiaming Zhou, Tianming Liang, Yi-Xing Peng, Yu-Ming Tang, An-Lan Wang, and Jonathan Roberts their valuable sugges- tions. The authors also thank Samuel Albanie, Chengke Bu, Tianshuo Yan, Yuxian Li, Zhiwei Xia, Zhi Ouyang, Qiaochu Yang, Jialu Tang, Shiyang Chen, Hanyi Xiong, Shenru Zhang, and Boning Shao for their help and support for our project.
10

---

References
[1] Md. Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A comprehensive survey of deep learning for image captioning. ACM Compuring Surveys, 51, 2019.
[2] Matteo Stefanini, Marcella Cornia, Lorenzo Barakdi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From Show to Tell: A survey on deep learning based image captioning. IEEE TPAMI. 45, 2023.
[3] Young Kyun Jang, Junmo Kang, Yong Jae Lec, and Donghyun Kim. MATE: Meet at the embedding connecting images with long texts. In Findings of EMNLP, 2024.
[4] Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg. Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer. Su Wang. and Jason Baldridge. DOCCI: descriptions of connected and contrasting images. In ECCV, 2024.
[5] Lin Chen, Jinsong Li. Xiaoyi Dong. Pan Zhang. Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. ShareGPT4V: Improving large multi-mode models with better captions. In ECCV, 2024.
[6] Xiaotong Li, Fan Zhang, Haiwen Diao, Yueze Wang. Xinlong Wang, and Lingyu Duan. DenseFusion- Merging vision experts for comprehensive multimodal perception. In NeurIPS, 2024.
[7] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Iyer, Sai Charitha Akula, Shusheng Yang. Jihan Yang. Manoj Middepogu, Ziteng Wang. Xichen Pan, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. In NeurlPS, 2024.
[8] Davide Paglieri, Bartlomiej Cupial, Samuel Coward, Ulyana Piterbarg. Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Lukasz Kucinski. Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker- Holder, and Tim Rocktäschel. BALROG: Benchmarking agentic LLM and VLM reasoning on games. arXiv preprint arXiv:2411.13443 2024.
[9] Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, and Tim Genewein. LMAct: A benchmark for co-tontext imitation learning with long multimodal demonstrations. arXiv preprint arXiv:2412011441, 2024.
[10] Weaye Lin, Jonathan Roberts, Yunhan Yang. Samuel Albanie, Zongqing Lu, and Kai Han. GAMEBoT: Transparent assessment of LLM reasoning in games. In ACI, 2025.
[11] Yunhao Gou, Kai Chen, Zhili Liu, Lanqing Hong. Hang Xu, Zhenguo Li, Dit- Yan Young, James T. Kwok, and Yu Zhang. Eyes Closed, Safety on: Protecting multimodal ilms via image-to-ten transformation. In ECCV, 2024.
[12] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xic. Eyes wide shut? Exploring the visual shortcomings of multimodal 1lss. In CVPR, 2024.
[13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishknn, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[14] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. PIXART-E: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In ECCV, 2024.
[15] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. /CML, 2023.
[16] Roopal Garg, Andrea Burns, Bureu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. ImagelnWords: Unlocking hyper-detailed image descriptions. In EMNLP, 2024.
[17) Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. CapsFusion: Rethinking image-text data at scale. In CVPR, 2024.
[18] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In /CML, 2021.
[19] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, 2016.

---

[20] Oriol Vinyals, Alexander Toshev, Sany Bengio, and Dumitru Erhan. Show and Tell: Lessons learned from the 2015 MSCOCO image captioning challenge. IEEE TPAMI, 39, 2017.
[21] Steven J. Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. CVPR, 2017.
[22] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. DenseCap: Fully convolutional localization networks for dense captioning. In CVPR, 2016.
[23] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image Captioning: Transforming objects into words. In NeurlPS, 2019.
[24] Lun Huang. Wenmin Wang. Jie Chen, and Xiaoyong Wei, Attention on attention for image captioning. In ICCV, 2019.
[25] Guang Li, Linchao Zhu, Ping Liu, and Yi Yang. Entangled transformer for image captioning. In ICCV, 2019.
[26] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memon transformer for image captioning. CVPR, 2020.
[27] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural image caption generation with visual attention. In ICML, 2015.
[28] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via visual sentinel for image captioning. In CVPR, 2017.
[29] Kishore Papineni. Salim Roukos. Todd Ward, and Wei-Jing Zhu. Bleu: a method automatic evaluation of machine translation. In ACL, 2002.
[30] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization @ ACL 2005, 2005.
[31] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. description evaluation. In CVPR, 2015.
CIDEr: Consensus-hased image
[32] Junhua Jia, Xiangqian Ding, Shunpeng Pang. Xiaoyan Gao, Xiaowei Xin, Ruotong Hu, and Jie Nie Image captioning based on scene graphs: A survey. Expert Systems with Applications. 231, 2023.
[33] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Ow: frontier large vision-language model with versatile abilities. arXiv preprint Xiv: /2966, 2023.
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. In NeurlPS, 2023.
[35] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The Ilama 3 herd of models. arXiv preprint 2rXi::240..21783, 2024.
[36] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In EMNIP,, 2021.
[37] Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara, Positive-augmented contrastive learning for image and video captioning evaluation. In CVPR, 2023.
[38] David M. Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, and John F. Canny. CLAIR: evaluating image captions with large language models. In EMNLP, 2023.
[39] Yebin Lee, Imseong Park, and Myungjoo Kang. FLEUR: an explainable reference- evaluation metric for image captioning using a large multimodal model. In ACL, 2024.
[40] Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du FAITHSCORE: Evaluating hallucinations in large vision-language models. arXiv preprint arXiv:231 21X00:2311.01447,, 2023.
[41] Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang. Yuan Zhang, and Haoyuan Guo. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405.19092 2024.
12

---

[42] Tianyu Yu, Haoye Zhang, Yuan Yao. Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He. Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. RLAIF-V: Aligning milms through open-source AI leedback for super GPT-4V trustworthiness. arXiv preprint arXiv:2405. 17220, 2024.
[43] Qinghao Ye, Xianhan Zeng, Fu Li, Chunyuan Li, and Haoqi Fan. Painting with Words; Elevating detailed image captioning with benchmark and alignment learning. In ICLR, 2025.
[44] Fan Lu, Wei Wu, Kecheng Zheng, Shuailei Ma, Biao Gong, Jiawei Liu, Wei Zhai, Yang Cao, Yujun Shen, and Zheng-Jun Zha. Benchmarking large vision-language models via directed scene graph for comprehensive image captioning. arXiv preprini arXiv:2412.08614 2024.
[45] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael Bernstein, and Fei-Fei. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 123, 2017.
[46] Hanoona Abdul Rasheed, Muhammad Maaz. Sahal Shaji Mullappilly, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Eric P. Xing, Ming-Hsuan Yang, and Fahad Shahbaz Khan. GLaMM: Pixel grounding large multimodal model. In CVPR, 2024.
[47] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. PixelLM: Pixel reasoning with large multimodal model. CVPR, 2024.
[48] Yugian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo. Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In CVPR, 2024.
[49] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu- Hong Hoi. Align before Fuse: Vision and language representation learning with momentum distillation. In NeurlPS, 2021.
[50] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Scyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. TMLR, 2022.
[51] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
[52] Kecheng Zheng, Yifci Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Drc Language-image pre-training with long captions. In ECCV, 2024.
[53] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit- Yan Yeung, Zhen Yang. Xiaodan Liang, and Hang Xu. Clip?: Contrastive manguage-imageppnn pretraining from real-world point cloud data, In CVPR, 2023.
[54] Yipeng Gao, Zeyu Wang. Wei-Shi Zheng, Cihang Xie, and Yuyin Zhou. Sculpting holistic 3D representa- tion in contrastive language-image-3D pre-raining. In CVPR, 2024.
[55] Jianzong Wu, Xiangtai Li, Shilin Xu, Haoho Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernar Ghanem, and Dacheng Tao. Towards open vocabulary learning: A survey. IEEE TPAMI, 46, 2024.
[56] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.
[57] Boyi Li, Kilian Q. Weinherger, Serge J. Belongie, Vladlen Koltun, and René Ranftl, I Language-driven semantic segmentation. In ICLR, 2022.
[58] Yu-Ming Tang, Yi-Xing Peng, and Wei-Shi Zheng. When prompt-based incremental learning does not meet strong pretraining. In ICCV, 2023.
[59] Ziyi Lin, Shijie Geng. Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang. Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen CLIP models are efficient video learners. In ECCV, 2022.
[60] Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. FROSTER: Frozen CLIP strong teacher for open-vocabulary action recognition. In ICLR, 2024.
[61] Kun-Yu Lin, Henghui Ding, Jiaming Zhou, Yi-Xing Peng. Zhilin Zhao Chen Change Loy, and Wei-Shr Zheng. Rethinking CLIP-based video learners in cross- cross-domain vocabuaayy action recognition. arXiv preprint arXiv:2403.0 240.005600 2024.
13

---

[62] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. In ICCV, 2023.
[63] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip H. S. Torr, and Song Bai. MOSE: A new dataset for video object segmentation complex scenes. /CCV, 2023.
[64] Tianming Liang, Kun- Lin, Chaolei Tan, Jianguo Zhang, Wei-Shi Zheng, and Jian-Fang Hu. Refer- DINO: Referring vidco object segmentation with visual grounding foundations. In ICCV, 2025.
[65] Jiaming Zhou, Ke Ye, Jiayi Liu, Teli Ma, Zifang Wang, Ronghe Qiu, Kun-Yu Lin, Zhilin Zhao, and Junwei Liang. Exploring the limits of Lision-language actiion manipulations in tross-task generalization. NeurIPS, 2025.
[66] Yi-Lin Wei, Jian-Jian Jiang. Chengyi Xing, Xiantuo Tan, Xiao-Ming Wu, Hao Li, Mark R. Cutkosky, and Wei-Shi Zheng. Grasp as you say: Language-guide dexterous grasp generation. In NeurlPS, 2024.
[67] Jiaming Zhou, Teli Ma. Kun-Yu Lin, Zifan Wang, Ronghe Qiu, and Junwei Liang. Mitigating the human-robot domain discrepancy in visual pre-training for robotic manipulation. In CVPR, 2025.
[68] Ke Ye, Jiaming Zhou, Yuanfeng Qiu, Jiayi Liu, Shihui Zhou, Kun- Yu Lin, and Junwei I. Liang. From Watch Imagine: Steering long-horizon manipulation via human demonstration and futurre envisionment. arXiv preprint arXiv:2509.22232 2025.
[69] Jiayi Liu, Jiaming Zhou, Ke Ye, Kun-Yr Yu Lin, Allan Wang, and Junwei Liang. EgoTra Beehh--BBhhh-e Towards robust trajectory prediction under ego-view noisy observations, 2025.
[70] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
[71] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. IntemVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint MxXiv:2312.14238 2023.
[72] Kunchang Li. Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. VideoChat: Chat-centric video understanding. arXiv preprint 2xXX:23305006355 2023.
[73] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng. Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. VisionLLM: Large language model is also an open-ended decoder for viston-centric tasks. In NeurlPS, 2023.
[74) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.
[75] Yi-Xing Peng, Qize Yang, Yu-Ming Tang, Shenghao Fu, Kun-Ye Yu Lin, Xihan Wei, and Wei-Shi Zheng. ActionArt: Advancing multimodal large models for fine-grained human-centric video understanding. arXiv preprint arXiv:2504.18155 2025.
[76] An-Lan Wang, Bin Shan, Wei Shi, Kun- Yu Lin, Xiang Fei, Guozhi Tang, Lei Liao, Jingqun Tang, Can Huang, and Wei-Shi Zheng. ParGo: Bridging vision-language with partial and global views. In AAAI, 2025.
[77] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Nie, and Ji-Rong Wen. A survey of large language models. arXiv preprint ar0iv: 2on.22333..2233. 18223, 2023.
[78] OpenAT. GPT-4 technical report. arXiv preprint ai 2023
[79] Wei-Lin Chiang, Zhuohan Li. Zi Lin. Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-souree chatbot impressing gpt-4 with 90% chatgpt quality, March 2023.
[80] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, arXiv preprint arXiv:2302. 13971, 2023.
14

---

[81] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv preprini arXiv:230..05277, 2023.
[82] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. Language models are few-shot learners. In NewrIPS, 2020.
[83] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
[84] Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang. Zhenhang Huang, Linjie Xing. Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, Yushi Chen, Tong Lu, Jifeng Dai, and Yu Qiao. The All-Seeing Project: Towards panoptic visual recognition and understanding of the open world. In ICLR, 2024.
[85] Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang. Qingyun Li, Lewei Lu, Xizhou Zhu, Yu Qiao, and Jifeng Dai. The All-Seeing Project V2: Towards general relation comprehension of the open world. In ECCV, 2024.
[86] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2.5-vl technical report, 2025.
[87] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: Reasoning segmentation via large language model. In CVPR, 2024.
[88] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos- 2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 148244 2023.
[89] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao. Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In ICLR, 2024.
[90] Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. Next-chat: An LMM for chat, detection and segmentation. In ICMI, 2024.
[91] Shilong Zhang. Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. GPT4Rol: Instruction tuning large language model on region-of-interest arXiv preprint arXiv:2307.03601, 2023.
[92] Alexander Kirillov, Kaiming Hc, Ross B. Girshick, Carsten Rother, and Piotr Dollár, Panoptic segmenta- tion. In CVPR, 2019.
[93] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[94] Jaemin Cho, Yushi Hu, Jason M. Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian Scene Graph: Improving reliability in fine-grained evaluation for text-to-image generation. In ICLR, 2024.
[95] Dahun Kim, Tsung- Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning open- woeld object proposals without learning to classify. RA-L., 7, 2022.
[96] Youcai Zhang. Xinyu Huang. Jinyu Ma. Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li. Shilong Liu, Yandong Guo, and Lei Zhang. Recognize Anything: A strong image tagging model. CVPR, 2024.
[97] Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari COCO-Stuff: Thing and stuff classes in context. In CVPR, 2018
[98] Shuai Shao, Zeming Li. Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019.
[99] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre- pre-training for open-set object detection. In ECCV, 2024.
[100] Akshita Gupta. Sanath Narayan, K. J. Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. OW-DETR: Open-world detection transformer. In CVPR, 2022.
[101] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, 2023.
15

---

[102] Maxime Oquab. Timothée Darcet, Théo Moutakanni, Huy V. Vo. Mare Szafraniec, et al. DINOv2: Learning robust visual features without supervision. TMLR, 2024, 2024.
[103] Jack Urhanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture is worth more than 77 text tokens: Evaluating CLIP-style models on dense captions. CVPR, 2024.
[104] Ronald Rensink, J Kevin O'regan, and James Clark. To perceive changes in scenes. Psychological science, 8, 1997.
to see: The need for attention
[105] Anne Treisman. Feature binding, attention and object perception. Philoxophical Transactions of the Royal Society of London. Series B: Biological Sciences, 353, 1998.
[106] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 2024.
[107] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, et al. Qwen2-VL: Enhancing vision-language model's perception the world at any resolution, arXiv preprint arXiv:2409. 12191, 2024.
[108] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, al. Molmo and PixMo: Open weights and open data for state-of-the thertrr multimodal models. arXiv preprint urXiv:2409.1711 2024.
[109] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVVsion Easy visual task transfer. arXiv preprint arXiv:2408.0332
2024.
[110] Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang. Zihan Liu, Jon Barker, Tuomas Rintamaki. Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NVLM: Open frontier-class multimodal 1lms. arXiv preprint arXiv:2409. 1440, 2024.
[111] 7he Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv preprint arXiv:2412.05271, 2024.
[112] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[113] Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Jong-CLIP: Unlocking the long-text capability of CLIP. In ECCV, 2024.
[114] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. TMLR, 2022, 2022.
[115] Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NV-Embed: Improved techniques for training LLMs as generalist embedding models. In ICLR, 2025.
[116] George A. Miller. WordNet: A lexical database for english. Communications of the ACM, 38, 1995.
[117] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[118] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summarics. In Text Summarization Branches Out, Barcelona, Spain, July 2004. Association for Computational Linguistics.
[119] Zhuang Li, Yuyang Chai, Terry Yue Zhuo, Lizhen Qu, Gholamreza Haffari, Fei Li, Donghong Ji, and Quan Hung Tran. FACTUAL: A benchmark for faithful and consistent textual scene graph parsing. In Findings of the ACI., 2023.
[120] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. CVPR, 2018.
[121] Pan Lu. Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song- Zhu, Oyvind Tafford, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NewrIPS, 2022.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Conclusion and Appendix sections, we discuss the limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results the paper the Method, Experiment and Appendix sections. We will release the code and data upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Readers can access our code and data via the link to our project page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss httppsccccpbccccccceececccccclcc htt.g/eeede/eeeeeeiiiaeeeeee /Cooeeeooiissseeeeeee wwssssnollicyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The paper specifies all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources in Ap- pendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://neett /ttpar/////eeip//eeeeee cc hwtpp//ww.....icccccscteeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses potential societal impacts of the paper in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly credits the creators original owners of all used assets, and properly respects the license and terms of use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our benchmark built upon the existing SA- IB dataset, and we provide details about benchmark in the Benchmark and Appendix sections. Our model is built upon the ASMv2-133 model, and we provide details about training and evaluation in Method, Experiment and Appendix sections. We will release benchmark and models upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We involve human labors in anontating test data and rating generated captions, and we include the instruction details for these two parts in Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There are no risks to human participants, and no risk disclosure or IRB approval is required.
Guidelines:
22

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: / www.e///eeeee//e/////// /LLM) for what should or should not be described.
23

---

Appendix
A Image-Text Retrieval Experiments
To demonstrate the application potential of our task and model, we apply our model to the downstream image-text retrieval task. In this experiment, we compare with two types of methods, namely image- text retricval methods and captioner- based methods, we first employ image captioners to generate the description for given query im- age, followed by retrieving similar descriptions using the NV-Embed- v2 text embedding model [115]. For models producing panoptic captions, we ask Qwen2.5-14B to transform panoptic captions into detailed natural- languagg captions without bounding boxes, NV- Embed-v2 exhibits superior perfor- mance on such descriptions. As shown in Table A1, on the chal- lenging DOCCI dataset, our Pan- capChain can achieve comparable performance with the state-of-the- art MATE model [3], despite using no image-text retrieval training data or specialized module designs. Our PancapChain also out- performs state-of-the-art image captioners (e.g., ShareGPT4V), demonstrating its effectiveness in capturing image details. In addition, using the Capture metrics [41], we demonstrate that PancapChain retrieves descriptions from the text corpus that are more semantically aligned with ground-truth de- scriptions, excelling on the dimensions of object, attribute, and relation. In summary, this experiment highlights the powerful application potential of our panoptic captioning model, as its straightforward integration with a text embedding model yields an effective image-text retrieval solution.
text-text retricval methods. Specifically, for captioner-
Table A1: Image-text retrieval results on DOCCI [4] in terms of Recall 1 and Capture metrics [41]. We report the similarity between generated and retrieved descriptions on the dimensions of object, attribute and relation using Capture. The bold num- bers indicate the best results.
Models
Type
R@1
Object
Auribute
Relation
CLIP [13]
Image-Tex Text
16.9
ALIGN [18]
Image-Ten Text
59.9
BLIP [15]
Image-Texe Text
54.7
LongCLIP [113]
Image-Texi
38.6
MATE [3]
Image-Text
62.9
GIT [114]
Text-Text
16.7
32.9
14.9
24.6
BLIP [15]
Text-Text
47.3
43.3
15.4
40.5
LLaVA-1.5 [74]
Text-Text
55.4
58.8
46,8
52.4
ShareGPT4V [5]
Text-Text
59.6
62.0
56.8
51.4
PancapChain (Ours)
Text-Text
61.9
65.2
60.3
53.6
B Image Reconstruction Experiments
To qualitatively demonstrate the effectiveness of model, conduct image reconstruction experiment by associating captioners with text-to-image generation models. This experiment serves proxy for evaluating the completeness of image descriptions, i.e., if a caption captures all essential visual elements, a text-to-image model would be able to reconstruct an image similar to the original one. In this experiment, we compare our PancapChain-I311 model with three baselines, namely Dwen2.5-VLL-22 [86], ShareGPT4V-133 [5] and BLIP-2 [15]. Specifically, our PancapChain and QWen2.5-VL. are instructed generate panoptic captions, ShareGPT4V generates conventional detailed captions, and BLIP-2 generates brief captions. Based on generated caption for an input image, we adopt the rext-to-image generation model PixArt-E [14] generate new image. Figure Al shows image reconstruction results using different captioners. As shown in the figure. PixArt-E associated with our PancapChain model can generate more similar images the original images, compared with other baseline models. For example, our model more accurately captures the person's leg the first sample, and it more accurately identifies the dog's location the second sample. Similarly in other examples, our model shows superiority in describing locations and attributes of primary entity instances images, i.e., the boy riding bike the third sample, the monk and cars in the fourth sample, and the men and horses in the fifth sample. Overall, this experiment demonstrates the effectiveness of our model in comprehensively capturing image details.
More Details about PancapScore
C.1 More Implementation Details
To evaluate models' performance in panoptic captioning, we propose a new metric named Pancap- Score, which comprchensively considers five dimensions of semantic content in panoptic captions.
24

---

Original Image
PancapChain
Qwen2.5-VL
ShareGPT4V
BLIP-2
EGS
Figure Al: Image reconstruction using PixArt-E [14]. Compared with previous models, our PancapChain can better capture image details, and thus lead to better image reconstruction. Best viewed in color.
PancapScore consists of three steps, namely semantic content extraction, entity instance matching and instance-aware question matching and instance-wware question answering.
question-answering answering. In this part, we introduce more details about entity instance
Entity Instance Matching, After extracting semantic content, we conduct an entity instance matching between the generated caption and reference caption to measure the performance in semantic tagging and instance localization. To find an optimal one-to-one matching for entity instances in the two captions, should assign semantic similarity score to each pair of predicted entity instance and ground-truth entity instance. Formally, the semantic similarity score is given as follows:
8i.j = p seq (3) + ssy (,) cos(!,, 1,),
(A1)
where μ 10 is weight coefficient raise the priority mentioned in main manuscript. In Eq. (A1), (4,, (2) is function that measures whether two tags are the same, i.e,, seq(4i, [,) if f; and E, use the same words. (t, is function that measures whether two tags share synonymous nouns, where we use WordNet [116] to get the synonym set of semantic tags for measurement. cos(4;, [2) denotes the cosine similarity between the two tags' text embeddings, where we encode the semantic tags using SeRtenceBERT 17]. We prioritize matching tags first by exact word matches, then by synonyms, and finally by embedding similarity, weighted by coefficient μ. Based on the assigned semantic similarity scores, obtain the optimal one-to-one matching
25

---

between two sets of entity instances. Since this entity instance matching is a bipartite graph matching problem, we solve it using the Hungarian algorithm.
Based on the entity instance matching, we measure the model performance in semantic tagging and instance localization. First, we focus on evaluating semantic tagging based on semantic consistency. Specifically, we consider two matched instances to be semantically consistent if their semantic tags use the same words, or have synonym nouns, have similar text embeddings, i.e., Si,i 3 6G. By considering all entity instances, we compute precision and recall in semantic tagging. Precision measures how many predicted instances have consistent tags according to the ground-trufe instances in the reference caption, while recall measures how many ground ground-truth instances are accurately recognized in the generated caption. Based on precision and recall, we compute the F-score for an overall measurement for semantic tagging.
Additionally, evaluate the performance entity instance localization based on location consistency. Specifically, we consider two matched instances to be consistent location, if they have consistent semantic tags and their bounding boxes have an IoU higher than a preset threshold, i.e., Sij > òe and iou(b, (b,, b,) > 8t. Similar to semantic tagging, we compute precision and recall in instance localization. Precision measures how many predicted instances have consistent locations according to the ground-truth instances in the reference caption, while recall measures how many ground-truth instances are accurately localized in the generated caption. Finally, based on precision and recall, obtain F-score in entity instance localization.
Overall, the proposed entity instance matching enables instance-level evaluation in semantic tagging and instance localization. is important to highlight that some previous metrics in conventional image captioning fail to distinguish different entity instances with identical semantic tags, or they attempt to distinguish them based on entity attributes [41]. Such solutions would easily cause inaccurate assessment of semantic tagging capabilities, as there may be numerous instances of the same tag in one image. Unlike these works, our PancapScore metric distinguishes entity instances by spatial locations, which is a more straightforward and accurate way.
Instance-aware Question Answering. Based on entity instance matching, we evaluate models' performance in attribute, relation and global state in instance matching provides a foundation for evaluating semantic tagging and localization, the question- answering approach offers more flexible framework to assess the more nuanced dimensions of attributes, relations, and global state. First, we generate questions based on the semantic content of the generated caption, where each question verifies whether attribute/relation/glol described in the reference caption. As each semantic item is associated with a unique entity instance, each instance ID in the generated caption should be mapped to the corresponding instance ID in the reference caption, as different captions assign different IDs to the same instance. For example, the generated caption contains an item "ID 2 is red" and the "ID 2" corresponds to the "ID 3" in the reference caption, we generate a question like "Is ID 3 red?". To improve the robustness of the evaluation, we generate a pair of questions for each semantic item: one with "Yes" answer and the other with a "No" answer.
-nwwwrinswwwgg manner. While entity
state item is
We employ a state-of-the-arr LLM to answer questions according to the reference caption. In this process, we instruct the employed LLM with a carefully designed prompt, and in-context text examples are included in the prompt to facilitate better understanding of our instructions. If the LLM outputs the same answer as the preset one, we consider the corresponding semantic item to be a correct prediction. By computing how many questions are correctly answered, we can obtain precision in predicting attributes, relations and global states. Similarly, we can measure recall on dimensions of attribute, relation and global state, where generate questions based on the semantic content of the reference caption and instruct the employed LLM to answer questions based on the generated caption. Finally, based on precision and recall, we obtain F-score on dimensions of attribute, relation and global state.
C.2 Human Consistency Analysis
In this part, we conduct an analysis to demonstrate the reliability of our proposed PancapScore metric. Following previous works [43, 41], we randomly sample 500 panoptic captions of test images in our SA- SA-Pancap benchmark, which are generated by different models. We then ask human annotators to rate each caption, where ratings are given on all dimensions for each image-caption pair. Specifically,
26

---

Table A3:
Statistics of the training. validation and test sets in our SA-Pancap benchmark. In the table, "Word", "Instance", "Attribute". "Relation" and "Global" denote the number of words, the number of instances, the number of attribute items, the number of relation items and the number of global state items mentioned in a caption on average. "Category" denotes the total number of entity categories in the whole set.
Word
Category
Instance
Attribute
Relation
Global
Train
257.5
2429
6.9
12.7
8.7
1.5
Validation
275.9
729
6.9
11.9
8.2
1.4
Tost
309.6
306
9.9
13.4
12.4
1.9
we ask human annotators to follow a clear and strict scoring process, designed according to our task formulation. First, we employ Qwen2.5-14P) to extract semantic elements from models' panoptic captions. Then, human annotators score each caption across all five dimensions, by leveraging the ground-truth panoptic caption reference. For each dimension in generated caption, they carefully check every element. a caption correctly identifies an ground-truth element in the reference, it gets point. If it misses or gets element wrong, it loses a point. In the end, human annotators combine the scores from all five dimensions to obtain an overall score for the caption.
After the human rating process, we compute the statistical correlation to compare the proposed PancapScore with human ratings. We use three metrics to measuring the correlation, including the Pearson correlation coefficient (PCC) p. coefficient of determination R and Kendall's 7 (Kd T). For comparisons, we se- lect three conventional captioning metrics (i.e., BELU, ROUGE and CIDEr) and one state-of- the-art open-source detailed captioning metric to construct baseline metrics, as panoptic cap- tioning is new task. Specifically, we first ask Qwen2.5-14P 14B to decouple a panoptic caption into two parts, namely a detailed textual de- scription without bounding boxes and a set of bounding boxes. Then, we evaluate the caption quality using these metrics and the localization capabilities by loU-based score. As a result, we can obtain four baseline metrics for panoptic captioning. As shown in Table A2, these baseline metrics exhibit limited effectiveness in human consistency. The performance of Capture, which depends on a text scene graph parsing model [1 [119] for semantic content extraction, is notably restricted in achieving human consistency and fails to fully account for all facets of semantic content, such as the global state. Conversely, our PancapScore metric obtains a substantial improvement over the baseline metrics in human consistency, underscoring its enhanced reliability.
Table A2: Correlation between panoptic caption- ing evaluation metrics and human judgements. The bold number indicates the highest human consis- tency among all caption metrics.
Metries
PCC (p) 1
I-R2 |
Kd r 1
BELU Box
0.02
16.03
0.02
ROUGE (118)-BoxScure
0.04
15.42
0.03
CIDEr [31 BoxScore
0.01
16.30
0.02
Capture [41 BBoxScrc
0.17
80.29
0.12
PancapScore
0.60
9.07
0.40
D More Details about SA-Pancap
Based on our proposed PancapEngine, we contribute a new SA-Pancap benchmark for the panoptic captioning task. We select the SA-IB dataset [101] as the data source due to its high image quality and data diversity. Overall, our SA-Pancap benchmark comprises 9,000 training and 500 validation images paired with auto-generated panoptic captions, and 130 test images paired with human-curated panoptic captions. Due to the challenging nature of panoptic captioning, we strategically select images containing fewer than 10 entity instances for the construction of SA-Pancap. Note that our PancapEngine has the capability to generate panoptic captions for images with a higher density of entity instances. As shown in Table A3, our SA- SA-Pancap demonstrates substantial diversity in entity category, and contains rich semantic content and high-quality comprchensive panoptic captions.
For the test set, we ask human annotators to refine model--genratee panoptic captions, following a rigorous curation process. Specifically, the curation process begins with the localization and identification of entity instances within a given image. Human annotators are first tasked with carefully reviewing and refining the bounding boxes and semantic tags associated with each entity instance. Following the refinement of entity instances, annotators proceed to enhance the descriptive details of each instance by refining their attribute and relation information. This involves a thorough examination and correction of the model-generated attributes (e.g., color and material) and the
27

---

relationships between different instances (e.g., action relation and part-whole relation). Next, the annotators proceed refine the global state information of the image. Finally, through the systematic integration of semantic information from all five dimensions, we derive high-quality panoptic captions that accurately and comprehensively describe the visual content. According to our entity detection suite, our entity instance annotation adopts a second-level entity granularity. For instance, as shown in Figure in our main manuscript, the 1-shirt is treated as single entity, and the text on the t-shirt is also recognized as distinct entity. However, we do not further decompose the text into individual characters, avoiding recursive subdivision beyond this level.
E Experiment Setups
E.1
Implementation Details
In this part, we present more details of our PancapChain model. Specifically, our model decouples the task into four stages, and constructs four types of image-prompt- prompt-text tuples for each image for training based on the ground-truth panoptic caption. We mix these four types of tuples together for training models, and we generate panoptic captions stage-by-stad loss is the standard auto-regressive loss (i.e., next token prediction) following L.LVV [34]. The data requirements and prompts of four model stages are as follows:
during inference. The training
Entity Instance Localization: Each training tuple in this part consists of image, prompt, and localization text. The localization text is in the format of
"<box> [Ø], yi, 222 //]</box>, <box> [oî, vi. 22, y0 </]o//ox>,... 222 [/]///>>""
<box> [xy, yf
where wi) and (1zy v²) denote the top-left and bottom-right corners of the bounding box of the i-th ground-truth entity the image. The prompt template for these data "Please localize all entities in this image".
Semantic Tag Assignment: Each training tuple in this part consists of an image, sample-specific prompt, and an instance text. The instance text in the format of
"+! <box> [at vi. xh. y/] ///x//ox>, t <box> Lof, vf. 22 //]</box>,... t <box> Lx yi, 222 y/]</box>",
where e is text describing the ground-truth semantic tag of the i-th entity in the image. The prompt template for these data is "Please specify the semantic tags of all entities based on their bounding boxes: {...}", where "(...)" includes the localization text the first stage, e.g., "<box> [100, 100, 200, 200x</box>, <box> [50, 100, 150, 300]</bor </bo>>,. <box> [90, 75, 500, 300] </bo>>".
Extra Instance Discovery: Each training tuple in this part consists of an image, a sample-specific prompt, and extra instance text. During training, randomly split the ground-truth instance set into two parts for an image. As a result, we can split the instance text into two texts, namely
"41 <box> [xt vi. 22, y ///</box>, 12 <box> [xf. vi. 22 //]< <//bboxx..... g n1 ni 11 <box> [x 31 x2 3/2 ]</bx>>"
and
"6l <box> [x], wi. xy, } </b>>> 1 <box> bof, v2, 2² 2 </bxx>, <box> [a 31 x2
n
3/ "1 <<box>>>>"
where nı + n22 = n. We use the first text as the extra instance text as output (for su- pervision), and use the second text to construct the input prompt. The prompt template for these data is "Please specify missing entities and their locations for this image based these specified entities: {...}", where "(. is filled in with the second text.
Panoptic Caption Generation: Each training tuple in this part consists of an image, sample-specific prompt, and a ground-truth panoptic caption. The ground-truth panoptic caption from our data engine. The prompt for these data is "Please provide a hyper letailed description
28

---

Table A4: Ablation study results the validation and test sets of SA-Pancap. SLoc Tag. Disc refer to our entity instance localization, semantic tag assignment and extra instance discovery stages, respectively. The caption generation stage Scap is required in all cases.
Validation
Test
Models
Tag
Loc
Att
Rel
Glo
All
Tag
Loc
Att
Rel
Glo
All
Baseline
56.38
29.30
43.06
31.64
82.47
168.63
55.12
30.00
43.09
28.88
78.91
164.98
w/ Instance Discovery
56.47 29.61
43.71
32.62
82.33
170.64
55.32
30.34
43.67
30.55
79.45
167.83
w/ Staee Tagl
57.04
29.83
43.76
33.69
84.16
172.74
55.92.
30.82
43.99
31.87
79.23
170.52
Full (w/ St Tag, Dise))
57.56
30.34
44.78
34.61 84.59
175.75
56.45
31.76 44.46 32.54
79.85
173.19
for this image, including all entities, their locations, attributes, and relationships, well as the global image state, based on boxes and tags: {...}", where "{...}" filled in with the complete instance text in the second stage.
Our model adopts the general LLaVA architecture [34], and it is initialized using the pre-trained ASMv2-13B [85] checkpoint, as it has good grounding capabilities. We finetune our model using LoRA (rank 128 and 2e-4). During inference, our model employs greedy decoding for caption generation. We train our model on the training set of our SA-Pancap for two epochs, and conduct evaluation on the validation and test sets. For our PancapScore metric, we use Qwen2.5- 14B [106] as the LLM for semantic content extraction and question answering. The threshold for measuring semantic consistency is set as 6, = 0.5, the threshold for location consistency is set as 6, = 0.5, and the weight coefficient Ag is set as 0.1. All experiments implemented using NVIDIA RTX A6000 GPUs.
256), and optimize using AdamW (batch size 128, learning rate
E.2 Baseline Setups
In our comparison experiments, we select nine state-of-the-arr MLLMs as baselines, including seven open-source large-scale MLLMs and two proprietary MLLMs. These models include Molmo-
72B [108], LLaVA-One Visi-onneiiioon- 72B [109], Qwen2-VL-72B
72B [110]. VntemVL-2..57788 [111], Llama-3.2-900 VL-72B [107], Qwen2.5-VL-722 [86]. NVLM-
[35]. GPT-4o [78] and (emini-2.0-Prool The cutoff date for our model comparisons is set as February 7th, 2025. We carefully design prompts for these MILMs unleash their panoptic captioning power, where in-context examples are introduced for better instruct-following For a fair comparison, these prompts explicitly specify attribute, relation and global state types, which are the same as that in caption generation of PancapEngine. In addition, we use two finetuned MILMS as baselines for panoptic captioning, which finetuned on our training set. We select VA-5--- [74] and ShareGPT4V-I3B [5] here, since they have good performance in previous detailed captioning works and have the same parameter scale and architecture as our model.
F More Ablation Study Results
In this part, we present more results of ablation study on both validation and test sets of SA-Pancap. As shown in Table A4, model performance gets gradually better with the addition of more stages, as discussed in our main manuscript. Additionally, the findings on validation and test sets are consistent. However, additional stages do not always improve performance the global state dimension, which relies on the overall image characteristics rather than specific entity instances. Since our proposed task decoupling approach primarily focuses on separating semantic tags, locations and other details of each entity instance, it results in modest improvements in the global state dimension. Overall, the results in the table demonstrate the effectiveness of our designs, and show that our work establishes a strong baseline for panoptic captioning to drive further advancements.
G Can Panoptic Captions Improve MLLMs?
To further demonstrate the effectiveness of our panoptic captions, in this part, we made an attempt to introduce panoptic captions in improving MILMs. Specifically, we generate panoptic captions
e wwwp:/lamma.cmm/oos//eeeddeedd powps//mmmmmmomoooo//alaaaa....
ppeeail.oommiddwwwgg-----setmmmm deepmindd haapam//ooogggllooooooi//eemnnnn
29

---

Table A5: Results of LLaVA-1.5 by introducing panoptic captions in the pretraining stage. The
results demonstrate the effectiveness of our panoptic captions in improving MLLM pretraining.
Models
Pretraining Data (# of Samples)
Instruction Tuning Data VizWiz ScienceQA
Ilava-1.5-7b-tora
L.L. (558K)
LLaVA-1.5
47.8
68.4
1.5--l.5-7 (Ours)
LLaVA-1.5 (464K) + Ours (94K)
LLaVA-1.5
49.4
70.2
1lava-1.5-7b-lon (Ours)
LLa (558K)+ + Ours (94K)
LLaVA-1.5
51.2
70.7
using our PancapChain-133 model for MLLM pretraining and demonstrate its effectiveness on downstream image question answering benchmarks. First of all, we apply our model on SAM and COCO datasets to generate panoptic captions, and we obtain 94K image-caption pairs. We mix these 94K data with LLaVA-1.5's pretraining data (about 558K) and construct dataset to pretrain LLaVA- model. Following the same pipeline as VLa [74], we pretrain the model and then finetune it using LLaVA-1.5 instruction data based on LoRA. Following LLaVA-1.5, 115, we evaluate our finetuned model on two image question answering benchmarks, i.e., VizWiz [120] and ScienceQA [121], and the results are summarized in Table A5. As shown in the table, our model can obtain notable performance improvement improvements of 3.4% and 2.3% on VizWiz and ScienceQA respectively, which demonstrates the effectiveness and superiority of our panoptic captions.
We further demonstrate that, when using the same amount of training data in pretraining as that in LLaVA-1.5, our generated panoptic captions can also benefit downstream VQA tasks. In this case, we randomly sample 464K data from LLa VA-15's s full pretraining data (558K) and construct new dataset consisting of 558K data (464K model. This new dataset has the same amount of data as that of LLa our model can obtain performance improvements of 1.6% and 1.8% on VizWiz and ScienceQA. respectively. These results demonstrate the effectiveness of our panoptic captions and confirm that our performance improvement does not merely stem from an increase in the amount of pre-training data. We believe our panoptic captions can lead to larger improvements with more data.
data plus 94K ours) to pretrain LLaVA-115 As shown in Table A5,
H Prompt Analysis for MILMs
In this part, we conduct an analysis of the employed prompts for Qwen2-VL-72B [107]. Specifically, in this experiment, we use three types of prompts to instruct Qwen2-VL- for generating panoptic captions, as shown in Figure A2. The results of the model using different prompts are summarized in Table A6. As shown in the table, Qwen2 Q2L72-V obtains limited performance in solving panoptic
Prompt-Ai Al Plese provide a description for this image You should specify the spatial il lacations of entities the bounding box coordinates in the tormut of "[lxt yl 22, yel The (xt. yt and (22, yi) indicate the top-left and bottom-right coordiretes respectively and the volune of 'x"", "yth, and yp" shoul be the ronge of (o. We nequire that the chordioote valued (0. () shoul be the top-lett of the image and the coordinate valued should the botton right the image.
Prampt-B: Please provide a very detailed description for this image. The descript should cantois all crune and antities, their spatial locations. attrisutes, relationshiss betweon pair oetifics. ottriiutts include but are not lnsited on entity's color, shape. state, type, adoition, b brightneds. time day. and any other nanable elements the scene, Note that your surput description should only conset of contents visbly preserted in the image, and you should avsid including any subjective interpnclations, guesses, or inferences aboat the context or intert ussod isted with content. spatial lncetiore, you must speoff the bounding box teordimates in the format of "Ikt y), 22, ye]] The (xt, yt) and (x2, yP) indicate the top left botten. right saordiathe respectively, and values "d"", "x"" coa-dincte valued
rendaring envinorment
par
including
initte the
conottions,
shodows
Y22 valued should
the range of [0. We right
be
left image
Here I provide you one tist esample for you to formoline the output formot This rext esample is not releted to the irput image ell. Pazze prodisce your description follow ng the format of thi examplle (...)
Plzase output the detailed description of image follsing the requrements.
Prompt-C:
should somantic elements their
very detailed desoriph the
pa
pair
tinr The attributtes noluse, but reictiench pa include, but ane not ind 1o apatial actior, part whole relstiorch anvrommnn the inage, including but not inited to The light ny conditiets, naeo, shidoes, brightness Time of day, and any other notable elements scane For spatial lacetions you must spxoff the bounding box ccerdinahes the format "[jxt yl, yell" The (xt. y3) (xi, y2) indicate top and bottion right cacrdnates respectively and volues of "xt", "yl". "22" "ya" should image
linited
coler shops, state, meterdl textire, and text rendering The
pain of ertifies cdidtion please describe the evenall
range 10, 13
(1. should image. Nate that sutput description ahould only canaiat of contents siably presented in the iage, and you should oveid including sub jubiective interpretations, gueeses, inferences about the context or intont ossociated with the content.
Breed an the se menttic elemorts you have identifred, pleass cutput o very deteiled desorption for this image. provide you produce your descripn cesrripion following
exomple
to formelis output evemple:
esomple is
inport image
Please dinectly surput the detailed descript on of this image following mo the above requinenents.
Figure A2: Three types of prompts for instructing Awen2-VL-722 to generate panoptic captions. As shown in the figure, in-context examples are included in Prompt-B and Prompt-C.
30

---

Table A6: Results of Qwen2-VL-72B using different prompts on the validation set of SA-Pancap. In this table, Qwen-A/B/C denotes the result of Qwen2-VL-72B using Prompt-A/B/C (as shown in Figure A2). We also include the results of combining Qwen2- Dwen2-VL-72B with our proposed PancapChain (i.e., DL- -hhiin) and our PancapChain- 13B model for comparison.
Models
Tagging
Location
Attribute
Relation
Global
Overall
Qwen2-VL-72B-A VL-72B-A
48,37
10.25
34,22
21.29
82.42
122.37
Qwen2-VL-72B-B
50.28
11.56
36.68
24.66
86.50
131.82
Owen2-V1.-2B--C
49.85
12.92
37.83
24.71
86.30
133.96
Qwen2-VL-72B-Chal
52.17
15.40
37.90
25.83
85.62
139.88
PancapChain-138
57.56
30.34
44.78
34.61
84.59
175.75
captioning, across all three cases. This demonstrates the limitations of current Multi-modal Large Language Models (MLLMs) in solving panoptic captioning tasks, as they cannot achieve satisfactory performance through simply prompt engineering. Additionally, we integrate Qwen2-VL--72B with our proposed PancapChain methodology by decoupling the caption generation process into multiple stages during Dwen2-VL-72BBs inference. As shown in Table A6, applying our method into Qwen2-VL-722 achieves substantial improvement over the model with alternative prompts, which demonstrates the effectiveness of our idea. Our PancapChain- 13B model still outperforms Qwen2-VL-72B VL-72B equipped with our idea, demonstrating the superiority of our proposed model and data engine.
I Examples
In this part, we show some examples demonstrate the panoptic captioning capabilities of our PancapChain model. As shown in Figure A3, our model generates panoptic captions that are largely accurate and satisfactory.
PancopChain-188 image close-u up view three apples resting clear gloss picte. applaa
pasitioned
the
The apple the left co0e 328, 435, 609]]//es is doep ned co with dightly lighter red spot near the top. The apple in the center ibond 412, 696: 6541]//bsw a mix of rec and green, with the red dnniiiting the top holf and the green the botton half The opple on the right choso cboso[446 [460, 273, 740. 4771)//bo is solid red color.
FORE
TCR
"hee plete diseellt 273, 984, 744 Vibeo The plate placed on dark brown woodon surface, which forms the bodiground of the image. The lightiin ar soft and diffused, creating a worm and inittig ermssphers. Thene one no hersh shoctwer and the ovenall brightness moderate, The coffness light. moad of image coln and with Fecus being netural beauity app
nest clear and has dightly textured surfoce,
day oppeara
either early morning or late aehernoon, gi the
PancapChain 138: This image
close up view
turties gotting condy beach.
The turtls in the foneground ihoeellle. 400, 605 6531))/bom in poe tioned on the left side the frome, its body ongled dightly towords the right: Its shell is light brown color with darker brown markings and its flipoers are darker shode brewn. turtle's located frome. dightly sbove turtie clse facing the right side of the frone, and its shall and flipprrs are similer color the first turtie.
viable, fooing the right side beckground
frome
fecus aot
right This
onsh further
beach, cleser the
sand tboee
image The 9960)//bw a light beige color, with grand texture suggesting
taker dips creaning daytinee There shadons, the overal meed of the calm and menene. The brightness the high. contributiig to the clear vabiiiit of the turlts and the sund.
the bright
PancopChain-188 image captures brown and whine standing a grassy (244
located
ondjjvbbse, central patches hindguartees white side the frome.
has body Trwards left
visible
The field, located en tbsooito, 472, 997, B31l)//bao,, covered in greer gress with potches of dirt and mud. bucket is visbble botton right comner of frome, positioned (boxl[178 736, [32])//bess,
In the background, there neversl trees One lorge pouthoned at dseei[112 189, 388, 451]] /iowo and mother is on the right aide, poath oned at 1611 107 996. 481]j-/buso. The trees have grear leaves and are partially obscured by the ooe A free shump near the fesce, poaitioned at ibawo]1924 553, 884, 5651)//ber.
lscated on the left side of the frome,
weedon force fence made verfical wooder posts and horizonta wooden rais.
the right side the frome posittred thsso[jole 465, 996, 504))//oon
The day is visible the top left comer of the frome and is light blue color; The lighting in image is bright even, auggesting pesceful
doytime There visbie shadows. The overal mood of image
Figure A3: Example outputs of our PancapChain-11 13B models. For clarity, we visualize the bounding boxes and semantic tags of identified entity instances for each image.
31

---

J Discussions and Broader Impacts
This work introduces panoptic captioning, novel task striving to seek the minimum text equivalent of images - an ambitious yet challenging goal. We take the first step towards panoptic captioning by formulating it a task of generating comprehensive textual description composed of five dimensions for a given image. To study this task, we contributed a evaluation metric, an effective data engine, high-quality benchmark, and novel decoupled learning method.
Our work reals that, despite remarkable progress in generating detailed textual descriptions for images, existing MLLMs still struggle with panoptic captioning, demonstrating their limitations in bridging image and text modalities. Also, our evaluation provides insights into the performance gap between open-source and proprietary models. By finetuning on high-qualing data, PancapChain- 13B model beats state-of oftte-ff-thearr MLLMs, e.g., VLternVL-2.5-777 potential of our task and model in bridging image and text modalities. Additionally, when paired with a text retriever, our model achieves superior performance in downstream image-text retrieval, demonstrating the practical utility of panoptic captioners. We believe that, developing more powerful panoptic captioners can benifit various downstream applications or learning tasks, e.g., image-text alignment and multi-modal understanding.
and Gemini-2.0% 2.0-Po, highlighting the
Despite our efforts, panoptic captioning still faces numerous unresolved challenges, and our current task formulation remains approximation, not yet fully achieving our ultimate conceptual goal of "minimum text equivalence". While our formulation is not the optimal one, it offers straightforward and reasonable starting point. Alternative formulations for achieving minimum text equivalence, e.g. using multiple bounding boxes with pose vectors to describe entity locations and states, remain underexplored and are left for future research. Nevertheless, our work is an important step towards the conceptual "minimum text equivalence" and provides a solid foundation for future development. We believe work will act as catalyst to accelerate the progress of developing textual representations for images.
Broader Impacts. First, we do not manually check every caption, there may be some socially biased content. However, our captions are generated by advanced MLLMs designed to align with human values, significantly reducing the presence of socially biased content. Second, work uses image data from the SA-1B dataset to establish our benchmark, inheriting certain societal impacts, such privacy concerns. The SA-IB dataset masks most human faces in images, offering degree of privacy protection. Users of our benchmark must adhere to the terms and conditions of the original data sources. Lastly, like many existing MILMs, our model could be manipulated "jailbroken" to produce unfair or inappropriate captions. This risk underscores the need to enhance MILMMs' robustness and ethical alignment, which will ensure positive impacts across diverse applications.
32


Reviews:
[{"review_id": "KN2Mh8fcxu", "reviewer": "Reviewer_tSAM", "summary": "This paper introduces panoptic captioning, a new task that aims to generate a single, comprehensive textual description of an image, covering all entities, their locations, attributes, relationships, and the global scene. To support this goal, the authors propose PancapScore, a metric that evaluates captions across five semantic dimensions; PancapEngine, a data pipeline that detects diverse image entities and generates detailed captions; and SA-Pancap, a benchmark dataset with high-quality training data and a human-curated test set. They also present PancapChain, a multi-stage captioning model that sequentially processes image content and achieves state-of-the-art performance, outperforming even larger proprietary models like GPT-4o and Gemini-2.0-Pro.", "questions": "1. The author stresses the model’s minimal nature in the abstract and introduction, yet no experiments are presented to confirm this claim. Is there any proof demonstrating that the caption generated by model is indeed minimal and cannot be further reduced?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.The paper is well-written and clearly structured, with intuitive visualizations that enhance understanding.\n\n2.The overall idea is sound, and the multi-step captioning framework is logically designed and well-motivated.\n\nWeaknesses:\n\n1.The experimental evaluation is somewhat limited, as it only tests on the authors' own benchmark. It would strengthen the paper to include results on standard image question answering benchmarks to assess whether the proposed method provides broader improvements.\n\n2.The benefit of the generated captions for pretraining is not demonstrated. It would be valuable to show whether incorporating these captions into pretraining improves downstream performance.", "was_revised": true, "final_justification": "The rebuttal provides new controlled experiments showing that panoptic captions improve downstream VQA performance and clarifies the role of “minimum text equivalence” as a broader research goal. While the positioning could still be refined, the task formulation, data engine, and evaluation metric are valuable contributions, and the additional results strengthen the paper’s practical relevance. I raise my rating to Borderline Accept.", "rebuttal": "We greatly appreciate the reviewer's recognition of our work's sound idea, well-motivated model design and clear writing. Below, we provide careful responses w.r.t. all the concerns and provide additional perspectives to better assess our paper.\n\nQ1. Results on image question answering benchmarks\n\nFollowing the suggestion, we generate panoptic captions using our model for MLLM pretraining and demonstrate its effectiveness on downstream image question answering benchmarks. We would like to clarify that, our model is specifically designed for panoptic captioning, which is substantially different from VQA and thus it cannot be *directly* applied on image question answering benchmarks. Therefore, to address the concern, we conduct experiments based on pretraining. Specifically, we apply our model on SAM and COCO datasets to generate panoptic captions, and we obtain 94K image-caption pairs (considering time and computation cost, we cannot generate more in the short rebuttal phase). We mix these 94K data with LLaVA-1.5's pretraining data (about 558K) and construct a dataset to pretrain LLaVA-1.5 model. Following the same pipeline as LLaVA-1.5 [58], we pretrain the model and then finetune it using LLaVA-1.5's instruction data based on LoRA. Following LLaVA-1.5, we evaluate our finetuned model on two image question answering benchmarks, i.e., VizWiz [R1] and ScienceQA [R2], and the results are summarized in following table. As shown in the table, our model can obtain notable performance improvements of 3.4% and 2.3% on VizWiz and ScienceQA respectively, which demonstrates the effectiveness and superiority of our panoptic captions.\n\n| Models | Pretraining Data | Instruction Tuning Data | VizWiz | ScienceQA | \n| :- | :-----: | :------: | :-----: |  :-----: | \n| llava-1.5-7b-lora | LLaVA-1.5 | LLaVA-1.5 | 47.8 | 68.4 | \n| llava-1.5-7b-lora (Ours) | LLaVA-1.5 + Ours | LLaVA-1.5 |**51.2**|**70.7**|\n|\n\nThe above results on image question answering benchmarks demonstrate the effectiveness of our panoptic captioning. Also, in our main manuscript, we have shown the image-text retrieval experiment to demonstrate our model's effectiveness and superiority compared with state-of-the-art image captioners (e.g., ShareGPT4V). Due to time and computation constraints in the short rebuttal phase, we can generate only 94K data for pretraining. We believe our model can lead to larger performance improvements by using more data, and we will provide these results in our revision.\n\n[R1] VizWiz Grand Challenge: Answering Visual Questions from Blind People, CVPR 2018.\n\n[R2] Learn to Explain Multimodal Reasoning via Thought Chains for Science Question Answering, NeurIPS 2022. \n\nQ2. Benefit of the generated captions for pretraining\n\nFollowing the suggestion, we generate panoptic captions using our panoptic captioning model for MLLM pretraining, and we find that using panoptic caption data in MLLM pretraining can benefit downstream tasks. As shown in the reply and table above (for the first comment), using panoptic captions for pretraining can improve LLaVA-1.5's performance on two downstream image question answering benchmarks, i.e., +3.4% on VizWiz and +2.3% on ScienceQA. This demonstrates the effectiveness of our model and the superiority of our panoptic captioning. Due to time and computation constraints in the short rebuttal period, we can generate only 94K data for pretraining. We believe our model can lead to larger performance improvements with more data, and we will provide these results in our revision.\n\nQ3. About the claim of minimum text equivalence\n\nWe would like to clarify that, we do not claim that the caption generated by our model to have already been minimal and cannot be further reduced. As stated in Line 28-29 and Line 34-35, we claim that achieving the minimum text equivalence of an image is an *ambitious yet very challenging* goal, and our work serves as the *initial effort* towards this challenging goal. \n\n- Finding the minimum text equivalence of an image aims to develop a concise textual description that comprehensively captures its essential semantic elements. To explore this ambitious goal and make the problem tractable, our work formulates panoptic captioning as the task of generating a comprehensive textual description for an image, which captures all entity instances, their respective locations and attributes, relationships among entity instances, as well as global image state. Such a formulation serves as a reasonable *approximation* to our conceptual \"minimum text equivalence\", which captures basic semantic elements for comprehensiveness while excluding less critical or subtle details for conciseness, as stated in Line 35-41. \n\n- Additionally, compared with prevailing captioning works (e.g., BLIP-2’s brief captions and ShareGPT4V’s detailed captions), our panoptic captioning *mainly differs in comprehensiveness*, as stated in Line 41-47 and Line 157-161. Specifically, previous works usually vaguely specify locations by pure words, and they cannot very accurately describe the position of an object using several words. In contrast, our work accurately localizes entity instances using bounding boxes, which offers an *effective and efficient* way to describe position and occupied region of an entity instance with only several coordinate numbers (i.e., offer an efficient way to comprehensively describe positions). Experiment results in image-text retrieval (Table 1 and 5) and text-to-image generation (Figure 1 and 4) demonstrate that our panoptic captioning model outperforms BLIP and ShareGPT4V due to its comprehensiveness."}, {"review_id": "onTjMwYbOl", "reviewer": "Reviewer_Hf9S", "summary": "The paper describes a framework for Panoptic captioning of images, ie, a comprehensive text equivalent of images which includes detailed semantic descriptions of all the objects, including the attributes and location boxes in the images and relation between objects. The paper describes PanCapEngine - a method to generate high quality, PanCapChain - a model to improve captioning and PanCapScore - series of metrics to score for panoptic captioning. Experiments have been shown on the generated dataset, SA-PanCap comparing to previous methods for image captioning, and the proposed method PanCapChain shows better results.", "questions": "1. In Section 4, an Entity Detection Suite is mentioned for detecting entities in the images for preparing the dataset. Whereas in Section 5, no method for entity extraction is mentioned. During inference using PanCapChain, what is the method for entity extraction?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper has the following strengths:\n1. The described problem of panoptic captioning is not well explored in literature. The paper has covered all the parts of the problem, including model, dataset and evaluation.\n2. Experiments on SA-PanCap shows that proposed method outperforms other LLM based methods. Ablation has been shown on different parts of the method.\n\nThe weaknesses are as follows:\n1. The problem of panoptic captioning looks similar to scene graph description of the image, where typically all the object and relations in the image are described. Example work:\nJunhua Jia, Xiangqian Ding, Shunpeng Pang, Xiaoyan Gao, Xiaowei Xin, Ruotong Hu, Jie Nie,\nImage captioning based on scene graphs: A survey, Expert Systems with Applications\nThe paper does not clearly mention how panoptic captioning is different than scene graph captioning.\n2. Experiments are shown only on SA-PanCap dataset. Authors should show experiments on atleast one more dataset like GCG [43] to show generalizability of the method.\n3. In Section 5, lines 342-345, the training loss and prompts are mentioned very briefly, without much explanation. Can authors give more details on the fine tuning step, data requirements for fine tuning and the prompts used for PanCapChain.", "was_revised": true, "final_justification": "Rebuttal helped in clarifying doubts on differences with scene graph methods and generalizability of the method.", "rebuttal": "We greatly appreciate the reviewer's recognition of our comprehensive coverage of the new task (including model, benchmark, and evaluation metric) and our model's effectiveness. In the responses below, we carefully address all the concerns and provide new perspectives to better assess our paper.\n\nQ1. Differences from scene-graph-based captioning [R3]\n\nOur work fundamentally differs from scene-graph-based captioning in both research objective and methodology. \n\n- According to Jia et al. [R3], scene-graph-based captioning aims to concurrently solve comprehensiveness and *controllability* issues of previous captioning methods (before 2023). Specifically, these methods first generate scene graphs and then produce captions based on structural scene graphs, thus leading to better controllability. However, their scene graph generation models are limited to recognizing predefined object and relationship categories, e.g., the widely used VisualGenome benchmark covers a limited set of 150 object categories and 50 relation categories. In contrast, our panoptic captioning requires generating a comprehensive textual description for an image, which captures all entity instances, their respective locations and attributes, relationships among entity instances, as well as global image state. Our work serves as the initial effort to seek minimum text equivalence of an image, an ambitious yet challenging goal, providing much more comprehensive understanding of the visual context than scene-graph-based captioning. To address panoptic captioning, we develop an effective data engine with a carefully designed detection suite, which leads to high-quality data covering 2400+ entity categories and a far broader range of relationships in free-from text. Please note that, *expanding the number of object and relationship categories of a benchmark is very challenging, which requires much more powerful data annotation process.*\n\n- Regarding methodology, scene-graph-based captioning methods rely on scene graph generation models built from object detection and relationship classification. These models struggle with complex scenes due to their restricted object and relationship category sets. Additionally, scene-graph-based captioning methods usually integrate various outputs to generate final captions, and may incorporate additional components (e.g., region captioning models). Thus, their performance may be limited by the caption integration process and additional models. In contrast, our panoptic captioning model is based on Multi-modal Large Language Models (MLLMs), which adopts a unified architecture and can directly output comprehensive textual descriptions without reliance on any other models. Also, we propose a simple yet effective method to improve panoptic captioning, which decouples the challenging task into multiple stages. Our experiments show that our PancapChain-13B model can beat state-of-the-art large open-source MLLMs like InternVL-2.5-78B, and even surpasses proprietary models like GPT-4o and Gemini-2.0-Pro. \n\nWe will add more discussions in our revision.\n\n[R3] Image captioning based on scene graphs: A survey. Expert Systems with Applications, 2023\n\nQ2. Experiments on GCG dataset [43]\n\nFollowing the suggestion, we evaluate our model in grounded captioning on GCG, aiming to demonstrate the generalizability of our panoptic captioning model. For an image, the grounded captioning task in GCG requires generating a brief grounded caption (averaging fewer than 30 tokens), which primarily describes salient objects, their bounding boxes and some object attributes (e.g., color). Since our model is specifically designed for our panoptic captioning and produces longer and more comprehensive captions than those grounded captions in GCG, we assess performance by measuring the extent to which our panoptic caption covers the GCG caption for a given image. We employ a variant of our proposed PancapScore as the evaluation metric, which calculates the proportion of correctly identified ground-truth semantic items using Recall instead of the F1-score. We report metrics for semantic tagging, location, and attribute dimensions, as well as an overall metric combining all three. As shown in the table below, our PancapChain model can outperforms Qwen-2.5-VL-72B by 10.03% in the overall metric, highlighting the effectiveness and generalizability of our model. \n|Models|Tagging|Location|Attribute|Overall|\n|:-|:-----:|:------:|:-----:|:-----:|\n|Qwen-2.5-VL-72B|58.10|30.60|48.24|136.93|\n|LLaVA-1.5-13B-tuned|61.46|31.09|49.58|142.13|\n|PancapChain-13B (Ours)|**64.00**|**32.09**|**50.87**|**146.96**|\n|\n\nQ3. More details on the finetuning step, data requirements for finetuning and the prompts\n\nSure, we present more details here. Specifically, our model decouples the task into four stages and constructs four image-prompt-text tuples for each image for training, according to the ground-truth panoptic caption. *We mix these four types of tuples together for training models (namely finetuning), and we generate panoptic captions stage-by-stage during inference.* The training loss is the standard auto-regressive loss (i.e., next token prediction) following LLaVA [31]. The data requirements and prompts of four model stages are as follows: \n\n- Entity Instance Localization: Each training tuple in this part consists of an image, a prompt, and a localization text. This localization text is in the format of \"\\<box\\>[$x^1_1, y^1_1, x^1_2, y^1_2$]\\</box\\>, \\<box\\>[$x^2_1, y^2_1, x^2_2, y^2_2$]\\</box\\>,... \\<box\\>[$x^n_1, y^n_1, x^n_2, y^n_2$]\\</box\\>\", where $(x^i_1, y^i_1)$ and $(x^i_2, y^i_2)$ denote the top-left and bottom-right corners of the bounding box of the $i$-th ground-truth entity in the image. The prompt for these data is \"Please localize all entities in this image\". \n\n- Semantic Tag Assignment: Each training tuple in this part consists of an image, a sample-specific prompt, and an instance text. This instance text is in the format of \"$t^1$ \\<box\\>[$x^1_1, y^1_1, x^1_2, y^1_2$]\\</box\\>, $t^2$ \\<box\\>[$x^2_1, y^2_1, x^2_2, y^2_2$]\\</box\\>,... $t^n$ \\<box\\>[$x^n_1, y^n_1, x^n_2, y^n_2$]\\</box\\>\", where $t^i$ is a text describing the ground-truth semantic tag of the $i$-th entity in the image. The prompt for these data is \"Please specify the semantic tags of all entities based on their bounding boxes: {}\", where \"{}\" includes the localization text in the first stage, e.g., \"\\<box\\>[$x^1_1, y^1_1, x^1_2, y^1_2$]\\</box\\>, \\<box\\>[$x^2_1, y^2_1, x^2_2, y^2_2$]\\</box\\>,... \\<box\\>[$x^n_1, y^n_1, x^n_2, y^n_2$]\\</box\\>\".\n\n- Extra Instance Discovery: Each training tuple in this part consists of an image, a sample-specific prompt, and an extra instance text. During training, we randomly split the ground-truth instance set into two parts for an image. As a result, we can split the instance text into two texts, namely \"$t^1$ \\<box\\>[$x^1_1, y^1_1, x^1_2, y^1_2$]\\</box\\>, $t^2$ \\<box\\>[$x^2_1, y^2_1, x^2_2, y^2_2$]\\</box\\>,... $t^{n_1}$ \\<box\\>[$x^{n_1}_1, y^{n_1}_1, x^{n_1}_2, y^{n_1}_2$]\\</box\\>\" and \"$t^1$ \\<box\\>[$x^1_1, y^1_1, x^1_2, y^1_2$]\\</box\\>, $t^2$ \\<box\\>[$x^2_1, y^2_1, x^2_2, y^2_2$]\\</box\\>,... $t^{n_2}$ \\<box\\>[$x^{n_1}_1, y^{n_2}_1, x^{n_2}_2, y^{n_2}_2$]\\</box\\>\" where $n_1+n_2=n$. We use the first text as the extra instance text, and use the second text to construct the prompt. The prompt for these data is \"Please specify missing entities and their locations for this image based on these specified entities: {}\", where \"{}\" is filled in with the second text. \n\n- Panoptic Caption Generation: Each training tuple in this part consists of an image, a sample-specific prompt, and a ground-truth panoptic caption. The ground-truth panoptic caption is from our data engine. The prompt for these data is \"Please provide a hyper-detailed description for this image, including all entities, their locations, attributes, and relationships, as well as the global image state, based on boxes and tags: {}\", where \"{}\" is filled in with the instance text in the second stage. \n\nWe have included some implementation details in our Appendix (e.g., prompts in Section E.1), and we will add more details in our revision. \n\nQ4. Entity extraction method for PancapChain\n\nWe use our PancapChain model to conduct entity extraction (without any other models), as illustrated in Section 5 Line 307-324. Specifically, our PancapChain model decouples the panoptic captioning task into four stages, and the entity extraction is accomplished by the first two stages. \n\n- In the first stage, we localize entity instances by prompting our PancapChain model to output a text in the format of \"\\<box\\>[$x^1_1, y^1_1, x^1_2, y^1_2$]\\</box\\>, \\<box\\>[$x^2_1, y^2_1, x^2_2, y^2_2$]\\</box\\>,... \\<box\\>[$x^n_1, y^n_1, x^n_2, y^n_2$]\\</box\\>\", where \"[$x^1_1, y^1_1, x^1_2, y^1_2$]\" denotes that there is an instance located in the bounding box with $(x^1_1, y^1_1)$ as its top-left corner coordinate and $(x^1_2, y^1_2)$ as its bottom-right corner coordinate. \n\n- In the second stage, we assign semantic tag to each bounding box by prompting our PancapChain model to output a text like \"cat \\<box\\>[$x^1_1, y^1_1, x^1_2, y^1_2$]\\</box\\>, dog \\<box\\>[$x^2_1, y^2_1, x^2_2, y^2_2$]\\</box\\>,... bag \\<box\\>[$x^n_1, y^n_1, x^n_2, y^n_2$]\\</box\\>\", where \"cat [$x^1_1, y^1_1, x^1_2, y^1_2$]\" denotes that there is a cat bounded by the first box. \n\nQ5. Some minor things on paper formatting\n\nThanks for pointing out. We will fix them in our revision. In addition, we would like to clarify that, the spider figure in Figure 1 aims to show our model's superior performance in panoptic captioning. The figure shows that three state-of-the-art MLLMs (e.g., Gemini-2.0-Pro) struggle with panoptic captioning, while our proposed PancapChain performs generally better with a significantly smaller model size."}, {"review_id": "taNZXYfTzV", "reviewer": "Reviewer_nxRe", "summary": "This paper introduces a new task called panoptic captioning, which aims to generate the minimal text equivalent of an image by capturing all semantically important content. To support this task, the authors propose: (1) PancapScore, a new evaluation metric that assesses semantic content across five dimensions: semantic tags, locations, attributes, relations, and global image states. (2) PancapEngine, a data pipeline that follows a detect-then-caption strategy to build a high-quality dataset, resulting in the proposed SA-Pancap benchmark. (3) PancapChain, a multi-stage model that improves captioning quality by decoupling the generation process into step-by-step reasoning.", "questions": "Please refer to \"weakness\".", "limitations": "Please refer to \"weakness\".", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n-\tThe paper introduces a novel task and a comprehensive framework including a metric (PancapScore), a benchmark (SA-Pancap), and a baseline method (PancapChain).\n-\tThe multi-stage design of PancapChain is well-motivated and the experimental results show its effectiveness in capturing fine-grained image content.\n\nWeaknesses\n\n-\tThe discussion on potential real-world applications of this new task is relatively limited. While current image captioning models can miss key details, it is unclear whether generating descriptions of all entities is always desirable. In practice, users may prefer concise, focused summaries, and it is often possible to extract more details through follow-up queries to a MLLM. This paper should clarify the use cases where panoptic captions are preferable.\n-\tSome images may include background or irrelevant elements, and generating panoptic captions in such cases could reduce clarity and distract from the main content.\n-\tThe paper mentions a four-stage process in PancapChain. However, a more detailed analysis of each stage's output would enhance transparency.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I would like to keep my score with borderline accept.", "rebuttal": "We greatly appreciate the recognition of our novel task, comprehensive framework, well-motivated model design and model's effectiveness. We carefully address all the concerns below. \n\nQ1. Discussions on potential real-world applications\n\nThanks for the comment. We would like to clarify that our work focuses on generating comprehensive panoptic captions for images, and our goal is independent of actual user demands, which may vary from user to user. We agree that, in some specific applications, users may prefer generating concise captions or are only interested in certain aspects of the image. However, a comprehensive description will satisfy different needs, as one can derive concise descriptions or specific user-preferred aspects from it. Meanwhile, we believe there are many real-world scenarios requiring highly comprehensive image descriptions, e.g., learning image-text alignment, cross-modal retrieval and vision navigation for the blind, which are valuable use cases for our model. Generating comprehensive panoptic captions is highly challenging, and we have demonstrated that state-of-the-art MLLMs struggle to solve this task effectively. To this end, our work aims to address this challenging panoptic captioning task. We will follow the suggestion to add more discussions about this in our revision. \n\nQ2. About background or irrelevant elements in images\n\nWe would like to clarify that, background and some small elements could be valuable for certain practical applications. For instance, when a user asks MLLMs to describe the background of an image, models that generate brief captions without addressing the background cannot provide accurate responses. In addition, our model is specifically designed to generate comprehensive panoptic captions for images, rather than concise captions focusing solely on prominent elements. By generating highly detailed captions, our model enables a wide range of outputs tailored to user needs, showcasing the versatile applications of our panoptic captioning.\n\nQ3. Detailed analysis of each stage's output of PancapChain\n\nFollowing this suggestion, we conduct a detailed analysis to each stage's output of our PancapChain model. Specifically, we use our proposed PancapScore metric to quantitatively evaluate output quality in five dimensions. If a stage's output lacks content for a specific dimension, we mark the corresponding metric as \"-\". Additionally, for outputs containing only bounding boxes, we assume the semantic tagging is always accurate. The results are summarized in the table below, and we have some observations as follows: \n\n- From the second and third rows, we observe that entity localization performance decreases after incorporating semantic tagging, likely due to instances where a correct bounding box is paired with an incorrect semantic tag.\n\n- From the third and fourth rows, we find that introducing the extra instance discovery stage improves the performance of both semantic tagging and entity localization.\n\n- From the last two rows, we find that the output of caption generation stage heavily relies on the output of its preceding stage, as the semantic tagging and entity localization performance of these two stages are identical. \n\nBy decoupling the challenging task into four stages, our model can obtain significant improvement over the baseline in terms of the overall metric.\n\n| Models | Tagging | Location | Attribute | Relation | Global | Overall |\n| :-| :-----: | :------: | :-------: | :------: | :------: | :------: |\n| Baseline | 56.38 | 29.30 | 43.06 | 31.64 | 82.47 | 168.63 |\n| Entity Instance Localization | - | 39.59 | - | - | - | - |\n| + Semantic Tag Assignment  | 57.04 | 29.83 | - | - | - | - |\n| + Extra Instance Discovery | 57.56 | 30.34 | - | - | - | - |\n| + Panoptic Caption Generation | 57.56 | 30.34 | 44.78 | 34.61 | 84.59 |175.75|\n|"}, {"review_id": "VUIkMbqvoA", "reviewer": "Reviewer_2FRX", "summary": "This paper introduces panoptic captioning, a challenging image captioning task that aims for minimum text equivalence of images by generating comprehensive textual descriptions. These descriptions encapsulate all entities, their locations, attributes, relationships, and global image state, emphasizing semantic entity instance-level details and location accuracy. The authors propose PancapEngine for high-quality data generation and PancapChain, a novel method that outperforms current MLLMs on the new SA-Pancap benchmark, particularly in location accuracy.", "questions": "1. Regarding weakness 1, please provide a more detailed discussion of the differences between your approach and methods like ImageInWords (IIW). Clarifying the specific advancements or distinct aspects of panoptic captioning beyond merely requiring bounding box information would strengthen the originality claim.\n\n2. In Table 4's ablation study, the results for the global state (Glo) metric are not presented, and only validation set results are shown. Could you please provide the global state metrics and results on the test set for this ablation study? This additional information would offer a more comprehensive understanding of the PancapChain method's performance and contributions.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The paper introduces panoptic captioning, a more challenging image captioning task that emphasizes semantic entity instance-level descriptions and location accuracy. I think this is helpful for advancing more precise image captioning.\n\n2. The proposed PancapChain method demonstrates significant improvements over current MLLM models on the SA-Pancap benchmark.\n\n## Weaknesses\n\n1. Originality: A core aspect of the paper is its emphasis on entity-aware prompting, utilizing a two-stage captioning process involving entity detection followed by entity-aware caption generation. While the paper distinguishes itself by requiring bounding box information, similar ideas and methods have been explored in prior work, such as ImageInWords (IIW) (cited in line 158). Although IIW's final captions do not include bounding boxes, its data annotation process also involves object detection to obtain bounding box information.\n\n2. Effectiveness of PancapChain method: While Table 3 shows PancapChain outperforming existing MLLMs on validation sets across most metrics except global state (Glo), its advantage is most significant only in location (Loc). This advantage in Loc is expected, as other MLLMs have not been trained on data explicitly including bounding box. Furthermore, this consistent advantage is somewhat diminished on the test set. Additionally, in the ablation study presented in Table 4, the numerical differences between PancapChain and the baseline appear to be modest.", "was_revised": true, "final_justification": "The authors' reply has successfully addressed my initial concerns, leading me to raise my score from borderline reject to borderline accept. Below are my initial concerns and how the authors address these.\n1. Originality: My initial concern was about the similarity between the proposed data annotation pipeline and that of ImageInWords. The authors' rebuttal successfully clarifies a significant distinction: while ImageInWords relies on a human-in-the-loop annotation process, their method is fully automated. This is a major advantage and a clear contribution.\n2. Method Effectiveness: The authors provided new experimental results demonstrating that their PancapChain method also shows improvements on Global metric and achieves performance gains on the test set. This evidence effectively validates the effectiveness of the proposed approach.", "rebuttal": "We greatly appreciate the recognition of our new task and our model's effectiveness. Below, we carefully address all the concerns and provide new perspectives to better assess our paper. \n\nQ1. Differences from ImageInWords's data annotation process [14] \n\n- Although our data annotation process shares some similarities with that of ImageInWords [14], there is an important difference between the two. ImageInWords proposes a *human-in-the-loop* framework to curate detailed image descriptions, relying on costly human labor. In contrast, our annotation process is *fully automated*, making it much cheaper and more efficient. As shown in our experiments, the superiority of our model (trained on automatically-annotated data) over state-of-the-art MLLMs demonstrates the high quality of our data. \n\n- Besides, we would like to clarify that, our contribution is not limited to a data engine for caption annotation. Overall, our work proposes a new task named panoptic captioning, serving as the initial effort to seek the minimum text equivalence of an image. Beyond our data engine, we contribute a comprehensive metric for reliable evaluation and an effective method to improve panoptic captioning. Experiments in image-text retrieval and text-to-image generation show our model's superiority over prevailing captioners (e.g., ShareGPT4V), highlighting the practical utility of our task.\n\n- Furthermore, we would like to emphasize that, introducing bounding boxes to describe locations is not trivial and offers significant value. In principle, by accurately localizing entity instances with bounding boxes, our formulation offers an effective and efficient way to describe position and occupied region of an entity instance using only several coordinate numbers. In contrast, prevailing captioning works, which vaguely specify locations by pure words, *cannot very accurately* describe instance positions. As shown in Table 1, our model outperforms the state-of-the-art detailed captioner ShareGPT4V in image-text retrieval. Also, it beats a state-of-the-art image-text alignment model ALIGN without requiring specialized training data or module designs. These results underscore the practical benefits of integrating bounding boxes in our task. \n\n- Last but not least, we would like to provide additional details about ImageInWords. ImageInWords does not open-source its models and training set, releasing only a small set of 400 images for evaluation. Based on our observation of these 400 released images, we find that these images usually features a limited number of object instances. In contrast, our dataset includes more complex scenes with a greater number of instances. As shown in Table 2, our captions are significantly longer, averaging 345.5 tokens compared to 217.2 tokens for ImageInWords, reflecting their greater detail and comprehensiveness. We will release our code, data and model. \n\nQ2. About performance improvement\n\n- First of all, we would like to clarify that, some MLLMs (e.g., Qwen-2.5-VL) involve training data explicitly including bounding box, e.g., grounding and region captioning data. In addition, we also compare our model with two baselines trained on our data. As shown in Table 3, our PancapChain outperforms LLaVA-1.5-tuned by 13.96% and ShareGPT4V-tuned by 19.05% in terms of the PancapScore metric on the validation set, which demonstrates the effectiveness of our model. \n\n- Second, we would like to clarify that our model can obtain substantial improvements *even without considering the location dimension*. The table below shows some model results in terms of an overall metric excluding the location dimension. As shown by the table, compared with state-of-the-art MLLMs, our PancapChain-13B performs much better with a significantly smaller model size, e.g., surpasses Qwen-2.5-VL-72B by 15.57% and Gemini-2.0-Pro by 11.32%. \n\n|Models (Val)|Tagging|Attribute|Relation|Global|Overall (w/o Location)|\n|:-|:-----:|:-------:|:------:|:------:|:------:|\n|Qwen-2.5-VL-72B|54.08|40.00|27.24|85.34|129.84|\n|InternVL-2.5-78B|54.68|41.81|27.41|88.37|132.74|\n|Llama-3.2-90B|52.87|39.94|27.09|83.40|128.25|\n|GPT-4o (Proprietary)|50.89|40.54|25.40|**88.85**|125.71|\n|Gemini-2.0-Pro (Proprietary)|53.79|43.14|28.52|86.50|134.09|\n|PancapChain-13B (Ours)|**57.56**|**44.78**|**34.61**|84.59|**145.41**|\n|\n\n- As shown in Table 3, our PancapChain obtains significant performance gains over state-of-the-art MLLMs on both validation and test sets. Notably, our model outperforms Qwen2.5-VL-72B (the strongest open-source MLLM here) by 26.21% on the validation set and by 18.86% on the test set. These performance gains are both substantial, as *the proprietary Gemini-2.0-Pro only outperforms Qwen2.5-VL-72B by 1.21% on the validation set and by 0.99% on the test set*. While there is some variation in performance gains between the validation and test sets, likely due to data distribution shifts, our improvement of 18.86% over Qwen2.5-VL-72B on the test set remains robust. Overall, PancapChain establishes a strong baseline for panoptic captioning and lays a solid foundation for future advancements in this field. \n\n- Finally, we would like to clarify that the performance gain of our model over our baseline is substantial. Specifically, in terms of the overall metric, our full model outperforms the baseline by 7.12% on the validation set. This improvement is notable, as *the  proprietary Gemini-2.0-Pro (which is expected to be much larger) only outperforms Qwen2.5-VL-72B by 1.21% on the validation set*. For more ablation study results, please refer to the response to the fourth comment. \n\nQ3. Differences from ImageInWords\n\n- Our work fundamentally differs from ImageInWords in both research objective and contribution. ImageInWords aims to improve detailed image captioning by a *data-centric* approach. Specifically, it carefully designs a *human-in-the-loop* framework to curate detailed image descriptions for model finetuning, relying on costly human labor. In contrast, our work proposes a new task named panoptic captioning, serving as the initial effort to seek the minimum text equivalence of an image – an ambitious yet challenging goal. We contribute a comprehensive metric for reliable evaluation, an *automated* data engine for high-quality data generation, and an effective method to improve panoptic captioning. Experiments in image-text retrieval and text-to-image generation demonstrate our model's superiority over state-of-the-art captioners (e.g., ShareGPT4V), highlighting the enormous application potential of our task. Furthermore, as shown in Table 2, our panoptic captions are significantly longer than those of ImageInWords, averaging 345.5 tokens compared to 217.2 tokens for ImageInWords, reflecting the greater detail and comprehensiveness of our panoptic captions.\n\n- Additionally, we would like to clarify that, introducing bounding boxes to describe locations is not trivial and offers significant value. In principle, by accurately localizing entity instances using bounding boxes, our formulation offers an *effective and efficient* way to describe position and occupied region of an entity instance with only several coordinate numbers. In contrast, prevailing captioning works that vaguely specify locations by pure words *cannot very accurately* describe instance positions. As shown in Table 1, our model outperforms the state-of-the-art detailed captioner ShareGPT4V in image-text retrieval. Also, it beats a state-of-the-art image-text alignment model ALIGN without requiring specialized training data or module designs. These results underscore the practical benefits of integrating bounding boxes in our task. We will add more discussions in our revision. \n\nQ4. More ablation study results \n\nFollowing the suggestion, we show more ablation results on both validation and test sets below, including metrics in all five dimensions. The notation aligns with that of our main manuscript. As shown in the tables, model performance gets gradually better with the addition of more stages, and the findings on validation and test sets are consistent. However, additional stages do not always improve performance in the global state dimension, which relies on the overall image characteristics rather than specific entity instances. Since our proposed task decoupling approach primarily focuses on separating semantic tags, locations and other details of each entity instance, it results in modest improvements in the global state dimension. Overall, the results in the tables demonstrate the effectiveness of our designs, and show that *our work establishes a strong baseline for panoptic captioning* to drive further advancements. \n\n|Models (Val)|Tagging|Location|Attribute|Relation|Global|Overall|\n|:-|:-----:|:------:|:-------:|:------:|:------:|:------:|\n|Baseline|56.38|29.30|43.06|31.64|82.47|168.63|\n|w/ Instance Discovery|56.47|29.61|43.71|32.62|82.33|170.64|\n|w/ $S_{\\\\{Loc, Tag\\\\}}$|57.04|29.83|43.76|33.69|84.16|172.74|\n|Full (w/ $S_{\\\\{Loc, Tag, Disc\\\\}}$)|57.56|30.34|44.78|34.61|84.59|175.75|\n|\n\n|Models (Test)|Tagging|Location|Attribute|Relation|Global|Overall|\n|:-|:-----:|:------:|:-------:|:------:|:------:|:------:|\n|Baseline|55.12|30.00|43.09|28.88|78.91|164.98|\n|w/ Instance Discovery|55.32|30.34|43.67|30.55|79.45|167.83|\n|w/ $S_{\\\\{Loc, Tag\\\\}}$|55.92|30.82|43.99|31.87|79.23|170.52|\n|Full (w/ $S_{\\\\{Loc, Tag, Disc\\\\}}$)|56.45|31.76|44.46|32.54|79.85|173.19|\n|"}]

Author rebuttal:

Meta-review: The paper proposes a novel task of panoptic captioning which involves providing a comprehensive caption describing all entities, attributes and relationships on the image. The contributions of the paper also include providing an automatic data engine and evaluation protocol for this new task. The model developed by the authors achieves strong performance, surpassing that of existing multi-modal LLMs. The reviewers raised several concerns regarding the motivation of such problem setup and its differences with existing tasks like scene graph generation, the value of the generated captions for pretraining vision-language models and the improvements of the proposed model beyond the benchmark introduced in the paper. Most of the reviewers' concerns were addressed during the rebuttal and discussion period. I recommend the acceptance for this submission and encourage authors to incorporate the results from the rebuttal into the camera ready version.
Decision: Accept (poster)