PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
Authors: Jang Hyun Cho
Andrea Madotto
Effrosyni Mavroudi
Triantafyllos Afouras
Tushar Nagarajan
Muhammad Maaz
Yale Song
Tengyu Ma
Shuming Hu
Suyog Jain
Miguel Martin
Huiyu Wang
Hanoona Abdul Rasheed
Peize Sun
Po-Yao Huang
Daniel Bolya
Nikhila Ravi
Shashank Jain
Tammy Stark
Seungwhan Moon
Babak Damavandi
Vivian Lee
Andrew Westbury
Salman Khan
Philipp Kraehenbuehl
Piotr Dollar
Lorenzo Torresani
Kristen Grauman
Christoph Feichtenhofer
Venue: NeurIPS 2025 spotlight ()
Keywords: llm
mllm
vlm
vision and language
perception
Primary area: applications
TL;DR: Perception Language Model

Abstract: Vision-language models are integral to computer vision research, yet many high-performing models remain closed-source, obscuring their data, design and training recipe. The research community has responded by using distillation from black-box models to label training data, achieving strong benchmark results, at the cost of measurable scientific progress. However, without knowing the details of the teacher model and its data sources, scientific progress remains difficult to measure. In this paper, we study building a Perception Language Model (PLM) in a fully open and reproducible framework for transparent research in image and video understanding.  We analyze standard training pipelines without distillation from proprietary models and explore large-scale synthetic data to identify critical data gaps, particularly in detailed video understanding. To bridge these gaps, we release 2.8M human-labeled instances of fine-grained video question-answer pairs and spatio-temporally grounded video captions. Additionally, we introduce PLM–VideoBench, a suite for evaluating challenging video understanding tasks focusing on the ability to reason about ''what'', ''where'', ''when'', and ''how'' of a video. We make our work fully reproducible by providing data, training recipes, code & models.

P:rceptionLM: Open-Access Data and Models for Detailed Visual Understanding
Jang Hyun Cho Andrea Madotto"., Effrosyni Mavroudi". Triantafyllos Afouras!." Tushar Nagarajan'* Muhammad Maaz** Yale Song",, Tengyu Ma Shuming Hu Suyog Jain', Miguel Martin', Huiyu Wang', Hanoona Rasheed Peize Sun', Po-Yao Huang', Daniel Bolya', Nikhila Ravit, Shashank Jain', Tammy Stark*, Shane Moon', Babak Damavandi", Vivian Lee', Andrew Westhury Salman Khan', Philipp Krähenbühhl, Piotr Dollár', Lorenzo Torresani"", Kristen Grauman 12,+ Christoph Feichtenhofer
'Meta UT Austin 'MBZUAI *Meta Reality Labs
*Joint first author
Work done during internships at Meta
*Project lead
Abstract
Vision-language models integral to computer vision research, yet many high- performing models remain closed-source, obscuring their data, design and training recipe. The research community has responded by using distillation from black-box models to label training data, achieving strong benchmark results, at the cost measurable scientific progress. However, without knowing the details of the teacher model and its data sources, scientific progress remains difficult to measure. In this paper, study building Perception Language Model (PLM) in fully open and reproducible framework for transparent research in image and video understanding. We analyze standard training pipelines without distillation from proprictary models and explore large-scale synthetic data to identify critical data gaps, particularly in detailed video understanding. bridge these gaps, we release 2.8M human- labeled instances of fine-graine video question guestion-answer answer pairs and tpatoo-temporally grounded video captions. Additionally, we introduce PL.M-Viee V.MM-VideBBech,, a suite for evaluating challenging video understanding tasks focusing on the ability to reason about "what", "where", "when", and "how" of a video. We make our work fully reproducible by providing data, training recipes, code and models at https: //gi /ttpttggttttttt/////gg www.cmmm//aaa/ce com/faacbbooressarcch. httpoooorreeeerrreeerrree..ooooo mmddels.
Introduction
Vision-language models (VLMs) are now key part of computer vision research and widely used in both academia and industry. Many of the strongest performing VLMs are closed-source, meaning their design, training methods, and the data they use are not publicly shared. To stay competitive, the research community has started to catch up to the proprietary models by using a straightforward approach distillation from black-box models [1-5], where proprictary models are directly used to label training data [3, 6, 7], directly leading to strong benchmark results.
Although distillation will unlock strong performance, there First, it makes hard to track scientific progress. Specifically, we cannot tell if better results on benchmarks are due advances in model design or training, or simply because the proprietary teacher models were trained on the evaluation sets of widely used benchmarks or internal data collected to resemble them this information is not available. Second, the heavy reliance distillation leads to fundamental misunderstanding of the effectiveness of current methods for training VLMs from scratch. Several key questions remain unanswered, including the significance of each training stage,
main issues for basic research.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Fine- line-Grained QA (FGQA)
Spatio-Temporal Captions (STC)
Perception Language Model
Question
The dng oakoua the persor throwing
Perception Encoder 0.30 /2
srich direction doss the persn fold the paper?
Liama 3- 5-18/ /B 18/99
Answer
Fiegonel puiling
dog of frrru
Image & Multi-image & Video
Largest video-language data
cantler of in undle crwwwe ofthe yellow paper.
The ding trings back to person.
Reproducible training
Figure 1: We introduce the largest collection of manually annotated fine- siue-graieed activity QA and spatiotemporal captioning data (left panel). Together with this data, we train and release PLM open and fully reproducible models to facilitate research in vision-language model training (right panel).
the influence of synthetic data, the data gaps that the research community should prioritize, and which of these gaps are currently being artificially addressed by distillation from proprietary models.
To better understand these challenges, we develop the Perception Language Model (PLM), a fully open and reproducible model for transparent research in image and video understanding (Fig. 1 right). PLM consists of a vision encoder with a small scale (<8B parameters) LLM decoder. We start by an analysis of standard training pipelines with available data, without any proprietary model distillation. We investigate large-scale synthetic data and establish key scaling laws to identify critical data gaps that limit video understanding performance, especially for spatio-temporal reasoning and fine fine-grained understanding tasks.
To fill these gaps, we create 2.8M high-quality human-labeled instances of fine-grained video QA and grounded video captions, see Fig. 1. This release is nearly an order of magnitude larger than the largest existing video datasets of each type [8, 9]. Our model, dataset and benchmark push the boundaries of video understanding, and provide a foundation for reproducible and transparent training and evaluation of VLM research. Across 40 image and video benchmarks, we achieve compa- rable performance with existing state-of-the-art open-weight models (e.g., Intern VL2.5 [10]), without distilling from proprietary models, and greatly outperform fully open models (i.e., Molmo (11)-
2 PLM: Overview
In this section, we overview the model, training stages and training data involved in the development of PLM. Please refer to Fig. 8 for a detailed overview and Appendix A for additional details.
Model. PLM consists of a vision encoder and language decoder, where a pre-trained Per- ception Encoder (PE) [12] is connected to the Llama [[3] language decoder (IB, 3B, or 8B parameters) with a 2-layer MLP projector. We use PE L/14 for Llama3.2 1B and 3B, and PE G/14 for Llama3. 1 8B. For image input, PLM incorporates dynamic tiling to support high res- olution images for up to 36 tiles of 448 reso- lution, where each tile undergoes 2 average
Stage
Stage
Stage 3
Warmup
Midtraining
Modality
Image
Image
Video Image + Vidco
Data
IM Synthetic
72M Mix
19M Mix
Training
Projector
Full
Full
Downsampling
2 2
2 x 2
Tiles/Frames
1/-
16/16
36/32
Table 1: Summary of three training stages to train PLM. See Appendix Table 8 and Table for data splits. pooling to compress the visual tokens. For video input, PLM uses 32 frames at 4482 resolution, where the same pooling is applied across the spatial dimensions of each video frame.
The data used to train the PLM consists of synthetic and human-ammotated samples. Synthetic data enhances the general capabilities of PLM, while human--nnotann annotated data broadens these capa- bilities to encompass more complex tasks. Synthetic data is sourced from diverse array of image and video datasets, covering fundamental VLM capabilities such as OCR, thart/dccumen/diagrua understanding, image/video captioning, and visual question answering.
Data.
We design data engines for each data modality (e.g., natural images, charts, documents, figures, egocentric and exocentric videos) to efficiently scale up, creating ~66.1M samples ($3). The synthetic data can be noisy, but is available at large scale; on the other hand, human-annotated data provides rich, high-quality supervision for image and video tasks. Here, we combine existing human annotations of diverse image and video sources, with our own collected annatttnd data, specifically geared towards grained video understanding and -patio-temporally grounded reasoning ($4).
2

---

Training stages. PLM trains in three stages:
1. Projector warm-up. First, we freeze the vision encoder and LLM and only train the vision projector on a small amount of synthetic image data. This warms-up the newly initialized parameters in the projector and improves stability for later stages. We use 1 M images from SA-1B [14] with the image captions generated by our data engine ($3).
2. Large-scale midtraining with synthetic data. Next, we train PLM on diverse domains of images and videos at scale, using a maximum of 16 tiles for images and 16 frames for videos. PLM sees around 64.7M images and videos with synthetically generated captions and question- answer pairs. We employ our data engine to scale up synthetic data generation (see $3).
Samples
Type
Stage
Our Haman-amnotated
(2,87M)
PLM-FGQA
2.4M
Fine-grained
3
PLM STC
476.2K R(D)Cap RTL
3
Our Synthetic (66.IM)
Natural Images
15.9M
Caption
1,2.3
Charis Docurnents
31.9M
Caption
2,3
Videos Mix
17.5M
Mix.
2,3
Ego4D
880K
Cap.
2,3
3. Supervised fine-tuning with human-annotated data. Finally, we train PLM with higher image resolutions and more video frames, using up to 36 tiles for images and 32 frames for videos. In this stage, we tackle more chal- lenging video tasks, including fine- QA and spatio- temporally grounded reasoning.
Exisring Open Source (6.52M)
Image (92 datasets)
5.6M
Diverse
2.3
Video (27 datasets)
920K
Diverse
2.3
Table 2: Summary of the data mix for train- ing PLM. See Table 10 for the full data blend.
Table shows an overview of our training setup for each stage. Appendix A. provides the complete training recipe for each stage, including hyperparameters and data sources. The compute cost of each PLM training stage is detailed in Tab. in the Appendix.
Synthetic Data Generation and Scaling
The predominant paradigm for VLM training is to generate synthetic annotations as cheap alternatives to human-labeled data [1, 10, 11, 15-18]. Although seemingly promising to get the best results on benchmarks, the majority of such data shared in the community is derived from proprietary models. This trend makes it hard to decouple scientific progress from proprietary distillation impact. In this section, we explore the efficacy of the current paradigm for VLM training in transparent manner. We design our data engine entirely from open-source models and scale the synthetic data generation to around 66. IM samples of images and videos. We establish the scaling laws of training from synthetic data on standard VLM tasks, including image, OCR/document, and video tasks.
3.1
Data Engine
Our data engine is designed to target base capabilities of VLMs for image and video understanding.
Image Data Engine. We generate short and long captions, as well as question-answer pairs, for natural images and those containing documents, diagrams, and text recognizable by optical character recognition (OCR). We prompt openly accessible Llama 3 [13] model to produce factual, detailed image captions while minimizing hallucinations. To create informative question-answe pairs, utilize OCR data, captions, and other metadata, which are fed into the prompt of a text-only LLM.
off-the-shelf scene detector [19] to extract video clips of approximately 30 seconds duration. Then, we extract the keyframes and generate frame-level captions using Llama 3, and video captions using our initial PLM trained with Stage 1 and Stage 3 data as shown in Table 2. We then employ an LLM to refine the frame-level and video captions by incorporating existing video metadata (e.g., action labels, time tags) into a cohesive, detailed video-level caption. Similarly, we generate question-answer pairs from the video-level captions.
Video Data Engine. For videos, first
The resulting synthetic data is large-scale and diverse - 66. IM samples carefully curated from variety of image and video sources including natural images, in-the-wild text, chart, figures, documents, egocentric and exocentric videos. Additional details are in Appendix I.
3

---

3.2
Scaling Laws with Synthetic Data
We examine scaling properties of our synthetic data under controlled setup and establish scaling laws.
Video QA Err. = (25e+03 x GFLOPs)-011
OCR QA Err.= [6.5e+03 x GFLOPs)-122
Natural QA Err.= (6.3e+02 x GFLOPs)
55
No Syth.
50
Syth. (1B)
45
No Syth (30)
No Syth. (38)
40
No Syth. (60)
No Syth. (6B)
Syth. (1B
35
No Syth. (38)
30
No Syth. (6B)
25
10TH
10
1044
--Power Faw
GFLOPs
Figure 2: Synthetic Scaling Plots. Relationship between Average Error across benchmarks and training compute (in floating-point operations) for various PLM models. We report average errors across Video QA tasks [8, 20-24], OCR QA tasks [25-28], and Natural Images tasks [29-34]. Model's performance using only human-labeled data subset are reported (No Syth.) as well the actual power-law fit of cach subcategory.
Setup. To establish power-law relationship between compute and validation-set errors of downstream benchmarks, we vary the scale of synthetic data, language model decoders (IB, 3B, and 8B), vision encoders (300M and 2B), and resolution/number of frames. For each configuration, we train a model with the MM synthetic data from our data engine and 6.5M publicly available human-labell following stage training described in $2. At every 2M samples, we evaluate PL.M on three categories of downstream benchmarks (VideoQA, OCR QA, Natural QA), constructed from 20 vision- language understanding benchmarks that provide a comprehensive and general evaluation of mudii- large language models. We compute the pareto frontier of these data points and fit a power law relationship: Err. = (3 x FLOP) and compare the exponents a of the power function as scalability of each setup, where smaller a implies better scaling.
data,
Scaling with decoder size. Fig. 2 shows the scaling behavior of PLM across various LLM sizes. We show validation-set errors and training compute on logarithmic scale, with the black linear line representing the power-law relationship between them. Different colors (green, turquoise, and blue) represent different language model scales (IB, 3B, 8B) while keeping the vision encoder size constant at 300M. As described in the setup section above, we show the power law fit of the pareto frontier in each benchmark category. We also show the results of PLM only trained on 4M numan-labeled datasets as baselines, denoted with horizontal lines of each color. The gap from the horizontal line to the data point marks the impact of the synthetic data. Interestingly, all three categories of benchmarks demonstrate clear power-law relationship between compute and average benchmark errors, with the power law exponent (a) of -0.15, -0.20, and -0.11 for Video QA, OCR QA, and Natural Image QA, respectively. In Appendix B, we provide more details and extend the analysis to (1) scaling the encoder size, and (2) scaling the image resolution and video frames.
Limitation of synthetic data. In Fig. 3, we evaluate stage 2 on an extended set of video benchmarks. Specifically, we show the result of 7 challenging video tasks on fine-grained activity understanding [35-39], temporal grounding [40] and long-video reasoning [41]. Unlike generic, high-level understanding (e.g., "what is happening in this video"). the "challenging" tasks require a thorough understanding of video in space and time, and fine-grained semantic details. As shown, the challenging video tasks ("HardQA" in lavender, plum, magenta) show a poor scaling trend (-0.03 (-0.03) compared to general video QA (-0.15). The stark difference between the two power law fits shows that scaling synthetic data is only effective for established, base tasks. Extending VLMS to these more challenging, complex tasks still remain unsolved. Next, we video data, PLM-FGQA and PLM-STC.
60
Avg. Errot
55-
Hard QA (140+02, 0.03)
50
45
Video QA (25 0.15
1012
-oower Fit
GFLOPs
Figure 3: Limitation of synthetic data. Challenging video tasks (HardQA [35-41]) do not scale well with synthetic data.
address this challenge with high-quality

---

Fine Fine-grained QA (FGQA)
Existing QA Detisets
FLM--IQOA jouns
Action Rec
fimettion
Courting
Ooj State
Question
Hicr dees the persons ho di the san ndwaper?
Question
how mang chakk amedda does the penson fiot
Pote
10x
Answer
Og Aurfouu
ald
Onj Location
oie oide, fingiem on the orive cats
conwer
Spacial Rel.
30x
Question
Question
Answer
voiih diveriion in thi person monsing the concippper?
Arsswer
Where is the meni siower lecored or the bing inninn???
Action Seq
ned liviing mott on
the
left humeer ofe portabll cinu..
posttioned
1C0K
200K
50CK
400K
Figure 4: Overview PLM-FGQA. Examples of inestiionanswer pairs from -GQAF focusing on fine- fine-grained human activity understanding. PLM-FGQA is approximately 8 times larger than the largest existing human-annotate video QA dataset and addresses a wide range of fine-grained question types that scarce in existing video QA datasets, such ones that cover direction of movement, object states, locations and spatial relations.
4 Human-annotated High Quality Data
As shown in Fig. 3, the current paradigm with synthetic data has run out of steam. Training from tens of millions of synthetically annotated data hardly improves our model on new, challenging video benchmarks. Beyond standard VLM tasks, these benchmarks focus on advanced capabilities such as fine-grained activity understanding. temporal grounding, and long video understanding. Perhaps, the knowledge that these benchmarks examine is simply not present the initial training set of our data engine nor in existing visual understanding to start from, that covers what, where, when, and how of activities in video. To address this gap, introduce two large-scale, human- annotated video datasets:
data. Our community lacks high quality datasets for detailed
PLM-FGQA fine-grand fine-grained video QA dataset collected by asking human annotators to watch a short video segment and answer model-generated questions which focus on "whar" activities humans perform and "how" they perform these activities. Question types include fine-grained recognition (action and object), fine-grained temporal perception (direction of movements, repetition counts, hand pose etc.), and fine-grained spatial understanding (object locations and spatial relationships). We use a multi-stage data engine to first extract video segments with salient actions from untrimmed videos through temporal clustering and shot-detection. Next, we generate questions and answers using either text-only LLM an early version of PLM. Finally, refine the answers by asking humans to verify or replace them if they are incorrect, resulting in a high-quality QA pairs.
Overall, we collect 2.4M question answer pairs from various open-access video datasets [42-47] spanning over 780k unique video clips from diverse domains (e.g., cooking, DIY, carpentry, auto- motive and bike repair) and viewpoints (egocentric and third-persone refer to Fig. 13 for domain statistics. This nearly 8 times larger than the size of the largest existing human-annotated video QA dataset in the community [48]. Moreover, as illustrated by the breakdown of question types' in Fig. 4 (top-right), PLM-FGQA contains a large number of annotations about fine-grained details that have been largely missing in existing training video QA datasets [24, 49-56]. Please refer to Table 19 for comparison with existing datasets Table 20 for dataset examples and Appendix F for further details.
PLM-STC is spatio-temporal video captioning dataset that offers detailed activity descriptions for each video. includes timestamps ("when"") of each activity and focuses on specific subjects identified by masklet ("where")). We employ two-stage annotation process to improve efficiency in collecting PLM-STC. In the first stage, annotators select interesting objects that exhibit significant motion changes in the video and use SAM 2 [57] to generate initial mask tublets, which they then refine to ensure high-quality spatial-temporal segmentation. For segments where the subject is out of frame, we automatically supplement "out of frame" caption. In the second stage, a separate set of annotators write temporally localized descriptions of the highlighted subject focusing on the changes in action across time in relation the whole video.
'obtained with LLM-based tagging.
5

---

Spatio-tempore Captions (STC)
10, TII Out framme.
10. 19 The man moves gracefully, using his hand gesturee closely resemble danie most octions.
[0, Ri] A litie gir moves back as beluge whale appreaches hor faco.
(12, 87) The person weer ing a jacket is running on snow covered ground. Sho stops and turns the other person.
(82, 85]
Out frame.
[20, 3i The person moves from right to
[IS, 98] She tries feed the whole
Figure 5: Overview of PLM-STC. Examples of spatio-temporally grounded captions from PLM- STC, the first dataset to associate each caption both with temporal interval as well as high-fps sequence of segmentation masks of the subject i.e., masklets (compared to just a temporal interval or sparse sequence of bounding boxes).
Overall, we collect 194.2K spatio-temporal captions the first existing large-scale dense video-region captioning dataset. We convert these spatio-temporal captions into three tasks for training: RCap (194.2K): Given the video region and timestamps, the model generates caption: RTLoc (194.2K): Given the video region and caption, the model localizes the action; and RDCap (122.3K): Given the video region, the model generates dense, localized captions. total, we construct 194.2K + 194.2K + 122.3K 522.7K samples, of which 476.2K are used for training and the rest for constructing PLM-VideoBench. Please refer to Fig. for dataset examples, Table 22 for comparison with existing datasets, Table 23 for dataset statistics and Appendix G for further details.
4.1 PeM-VideeBennh
Our high-quality human-annotated data offers VLMS train for broader range of capabilities for holistic video understanding. However, existing video benchmarks are not adequately equipped to evaluate these. To this end, introduce PLM-VideoBench, a novel benchmark focusing on specific activities (what) and their execution details (how) within spatio-temporal contexts (where and when).
Fine-Grained Question Answering (FGQA). In this task, model must answer multiple-choice question (MCQ) that probes nuanced, fine-grained activity understanding (e.g., painting "vertically" vs. "horizontally" in Fig. 6, first). We report multi-binary accuracy (MBAcc) [39] where each question is split into multiple binary choice questions. Our test set consists of 4,371 question-answe wew pairs. For more information, including statistics on video clips, segment duration, question types, and benchmark construction, see Table 21 and $F.2.
Smart Glasses Question Answering (SGQA). In this task, model must answer open-ended questions about activities and objects visible in an egocentric video stream recorded by a smart- glasses device (see Fig. 6, second). The questions are designed to simulate real-world scenarios where a user would ask for assistance from their smart glasses. We manually collect the videos using commercially available smart glasses, providing completely new, unique dataset that reflects modern use-cases such online Al video assistance and activity coaching. For evaluation, we use LLMe accuracy with an open-access model (Llama3.3 70B). The test set consists of 665 human-annotate question-answe answer pairs. See Appendix H for more details.
Video Region Captioning (RCap). this task, a model must generate a detailed description of an event involving subject of interest in the video. Given a region masklet and specified time interval, the model is required to output a caption that accurately describes the event occurring within that interval. Compared to traditional video captioning [47, 58, 59] where the aim is to generate video-level caption, the goal is to generate a region-level caption tied to a specific subject (e.g., a instances
person, object or animal) (see Fig. 6, third). The test set contains 10,060 and we report LLM-judge accuracy with Llama3.3 70B. See Appendix C.3 for details.
Region Temporal Localization (RTLoc). In this task, a model must identify the precise time interval within the video when the specified event takes place for the given subject. Given a video, region masklet and text description of the event, the model is required output the start and

---

FOQA
SGQA
RGap
RTLoc
RDCap
Question
Question ersimee not emouggh??
Question
Question
Question
mosing the painthouuh
netionn performen by the rege gireed poject
negiion whee does "The cat up front, trecertic the Vice" soour the visuo.
Inciicatidd thei notin mork with start and und frumos all actiorsid csscription of each action.
pamis
Choices
Answwer
Vartically from
hortom
pertectty eiminee, with shocss muter numoovdd
Answer
und
Answer
Answer
left thrnggt
trsoande car rea spitt aroundl
tnithe
Answer
mar driday bat
bussum
Figure 6: PLM-Video Dataset includes fine- fineggrained video QA (FGQA), open-ended QA in videos recorded using smart glasses (SGQA), Spatio-Temporal Captions (STC) post-processed into video region captioning (RCap), video region temporal localization (RTLoc) and video region dense captioning (RDCap) tasks.
end timestamps that correspond to the occurrence of the event (see Fig. 6 fourth). Notably, this task the inverse of RCap - instead of generating the caption, the model receives it as input and generates the corresponding time interval. We filter the test are unambiguously localized, i.e., they map to a single time window in the video. As a result, the test size is reduced to 7,910 instances compared to RCap. We report average recall @ over IoU thresholds (0.3, 0.5, 0.7, 0.9). See Appendix C.3 for details.
include only the captions that
Region Dense Video Captioning (RDCap). In this task, model must generate detailed description of all events involving a specific subject interest (e.g., person, animal, or object) video. Given video and region masklet, the model must produce sequence of (start, end, caption) tuples that cover the entire duration of the video, including periods when the subject is not visible (see Fig. 6, last). This task composition of RTLoc and RCap, requiring the model to produce both temporal windows for events as well as captions directly from the video. The test set contains 2,620 samples and we report the SODA score [60] which uses an LLM judge. See Appendix C.3 for details.
Experiments
We first overview the baselines and evaluation setting (85.1). We then compare benchmark results of PLMs with the baselines on a broad collection of image ($5.2) and vidco ($5.3) tasks as well as on ($5.4), Finally, we provide analyses data and model ablations ($5.5).
5.1
Setup
We compare PLMs against the following two classes of baselines:
Proprictary models such GPT-4o [61]
Gemini-Pro 1.5 [62] and Gemini-Flash 2.0 [63]. We use API calls to evaluate these models.
[64], Qwen2.5-VL. [15] and IL2e [10] - state-of-the-ee -hetthe-art open-acces. models, for which model scale, architecture and inference code are available. We use the official inference code for all models.
Open-access models such as Molmo-O [11],
Inference protocol. For mask inputs PLM- VideoBench, we overlay colored box on the video frames to specify the regions. We report validation set performance unless specified (in brackets) under the benchmark name. Metrics marked with use LLM as a judge. Complete implementation details including inference hypee-parametert task prompts, judge prompts and proprietary model evaluation protocol can be found in Appendix C.4.
5.2 Image Benchmark Results
We evaluate PLM on a total of 20 image benchmarks. Charts, Diagrams and Documents: answer questions that require parsing images documents and diagrams; Image Captioning: generate short/detailed caption, Perception and Reasoning: answer questions of varying difficulty about ob- jects, actions, functional correspondence, multi-view reasoning, spatial layout etc. and Hallucination: evaluate robustness to hallucinated details. More details in Appendix C.I.
7

---

Charts, Diagauns and Docameats
Percsptiin Reazosing
Hard Perceptton
Hollac
(anngee acc (70)
(000)) one (27)
(ool oo [68]
(vol) aoe [1)
couner imagc)
Besl Warks
Model
Doe
ChanQA
(65)
VOA
(66)
InfoQA
ADD
CWil most)
acc (67)
OCRDench
138)
MMMU
OK VO--
00C (60)
Via-We
(33)
SEED
[FL
CV OVAbeach
axc (P2)
129)
VSK
199)
HOPE
anc (32)
GPT-4o [6]]
92.8
85.7
75.3
80.7
94.2*
810
70.7"
63.9
77.1"
68.0"
72.5
73.9
78.0
87.2"
Gemini [63]
84.2
74.8
81.0"
63.2
77.8
99.8
76.1
88.2"
Gomini 2.0 Hosh [63]
93.0
8488
80.2
81.0
94.0
792
69
9
57.8
77.
64.4
82.3
71.9
74.8
1B scale
QwenVVL-2B [16]
90.1"
75,3
50.3
65.5"
84.6"
NO9
41.1"
80.0
59.7
67.4
72.9
44,4"
17.3
62.6
73.0
87.2
1. [10]
84.8
75.9"
72.0"
56.0"
40.9
72.2
71.3
42.4
58.3
65.4
90.2
PLM-1B
90.7
78.6
82.1
03.0
84.9
SUY
$4.3
$1.7
61.9
59.7
76.3
46.8
73.8
67.1
68.8
88.4
36 scale
Qwen2.5 3L [15]
$3.1
79.3"
77.1"
53.1
80.8
73.1
47.6"
56.4
78.5
88.2
L.sernVL2.5-68 [10]
91.6"
$4.0
79.3
72.1
90.5*
K28*
52.3
80.9
61.0
61.8
75.6
50.8"
55.9
64.6
80.0
91.0
PLM-B
84.3
R30
64.0
78.5
55.4
N0.4
88.7
8B scale
Molmo-78-O [11]
90.8
80.4"
80.4"
70.0"
90.7
39.3*
85.3*
67.5*
LLAVA-UN-78 [64]
56.7
77.3
08.8
656
48.9
K3.5
69.6
63.4
76.4
49.4
75.0
66.7
78.1
89.2
5VL [15]
95.7"
87.3"
84.9*
82.6"
93.0
864
58.6*
70.1
61.0
73.5
73.2
56.4"
11.9
69.8
80.3
87.2
1.serrVL2..588B [10]
93.0"
84.8*
79.3
77.6'
92.8*
523
56.0
80.6
69.2
64.3
77.6
54.8"
$3.9
70.1"
80.0
90.6"
PLM
35.5
86.5
80.9
92.7
46.1
85.6
69.6
79.3
56.0
82.8
89.9
Table 3:
Image benchmarks. PLM versus proprictary models and open-access baselines of comparable scalc. Cells with are reported numbers from literature, and the remaining are reproduced using official code.
VCig
Viden (A4
Fine-grained of Vidro Q4
T.far.
Hallar.
DREAM IK
FT [74]
MVVBauch
NE<TEQA
nce [50]
Poreeption Test
te oze [M4]
ace [31]
Video Male--MME MME
ncc [30]
Activtyy Nec QA ocet [5
Ego Schemma (eett) oce (22)
Tempori Bends
Mibicc (39)
TOMATO
(36)
MaticeBench
(deu ace [38]
TempC CompCoopucs
JMC 2 axe (75]
CG Bendh
(clac) acc (35)
Chandeo STA
mntOU! (40)
Videsdaalloo llace
overaal acc (76)
Ereat
(hinury) axe [37]
Model
STAR
Proprietary
GPT-4o [6]]
64.6"
79.1
70.4
71.9*
72.2*
38.5*
37.7"
55.9
74.5
58.3"
38.6
56.4
91.9"
Gemini [63]
81.6
65.9
36.7
71.2"
56.1
75.6
56.0
80,9
Gemini 2.0 Flach [63]
60.7
$1.9
20.3*
71.5*
27.6
32.8
56.1
76.9
47.0"
29.8
60.1
81.6
IB scale
QueaZVL-28 [16]
63.2"
76.4
53.9"
55.6"
58.4
27.0
25.7
46.9
62.3
42.8
0..
34.9
59.9
LmteraVL25:18 [10]
27.7
64.8
73..3
59.4
73.0
50.3*
60.7
55.7
27.7
25.0
45.0
56.4
40.9
0.8
31.0
38.9
PLM-ID
34.3
70.1
80.3
72.7
83.7
49.2
62.5
60.4
18.2
25,5
52.2
64.6
43.6
55.2
49.2
79.5
3B scale
Qwen2.5 VL-3B [15]
20.3
67.0
8
66.9
63.0
61.5*
59.2
64.8*
17.2
23.5
49.2
63.0
45.7
38.8"
45.2
53.5
1.25-1V....54BB [10]
20.2
71.7
82.5
67.9 9
77.2
62.3*
64.1 1
66.6
23.7
27.4
52.7
65.2
52.0
8.4
49.6
66.3
PLM-3B
74.7
83.4
79.3
54.9
66.3
66.9
23.4
30.9
60.4
69.33
47.2
57.7
55.5
76.5
LLAVA-OV-7B 1641
28.0
57.1
81.0
58.1
66.0
57.7
60.5
45.4
19.5
27.6
53.7
67.8
41.2
12.1
34,7
61.1
Qwen2 5VL-7B [15]
23.3
80.0
70.5*
63.7
65.0"
24.6
51.1
71.7*
50.1
152551VL...588 [10]
28.5
72.6
85.5
68.9*
77.6
64.2'
66 1
66.2'
24.3
20.4
53.5
68.3
53.1
14.3
57.1
60.2
PLM-SD
35.9
77.1
84.1
82.7
84.9
53.3
67.3
68.8
28.3
33.2
61.4
72.7
46.4
58.6
57.7
77.3
Table 4: Video benchmark results. PLM versus proprictary models and open-access baselines of comparable scalc. Cells with are reported numbers from litcrature and the remaining are reproduced using official code.
Table 3 shows our results. Overall, PLM shows strong performance on a wide spectrum of image benchmarks with solely from open-access data with white-box data engine. Additionally, we report Image Grounding task results on RefCOCO/+g [77] datasets in Appendix Table 15, and show that PLM outperforms both specialist models as well as the VLM baselines in all model scales.
5.3 Video Benchmark Results
We evaluate PLM on a total of 25 video benchmarks. We divide these into the following categories. Video Captioning: generate short caption for video, dense description of all events; Short video QA: answer a question about a short video (few seconds to a minute), either by selecting from a list of options, providing a free-form answer; Long video QA: answer question as before, about a much longer video (minutes to hours); Fine-grained QA: answer detailed questions about spatial location, motion, temporal information etc.; and Hallucination: evaluate the robustness of video models to hallucinated details about objects and events.
Table 4 shows video captioning, video QA, fine-graines video QA, and video hallucination results. We achieve strong results on widely adopted benchmarks, despite only using open-access data mix free from proprictary model artifacts, outperforming both the ocen-access and proprictary models.
8

---

Further, we achieve competitive performance on the majority of challenging benchmarks, such as EgoSchema (68.8 %). MotionBench (61.4 %). TOMATO (33.2 %). TempCompass (72.7 %). TemporalBench (28.3 &). Charades-STA (58.6 %), and more. All our model scales show strong performance against both proprietary models as well as open-access baselines of same scale.
Lastly, we also show that PLMs at all scale greatly outperform existing approaches on captioning tasks and hallucination detection tasks, owing to our focus on detailed, fine-grained spatio-temporal annotations in our human-annotated data collection.
5.4 PLM-VideeBench Results
We report the result on proposed benchmark LLMM-VieoBenc's from $4.1 in Table 5. We evaluate our PLM as well as (proprictary and open-access) baselines. In addition, we provide human performance of each subtask in the first row. The results show a significant gap between the baselines and PLM. Proprietary baselines and open-source baselines alike perform reasonably on FGQA tasks, though still 6.5 points lower than PLM (61.2 vs 67.7).
On SGQA, where the video sources and the question-answer pairs are unseen to all models, PLM performs reasonably well, yet 2.1 points short from open-access best (Intern VL2.5) and far from the best proprictary model (GPT-4o).
T00%
SGIA
RDCap
S0DM
v00mel
Model
Uoman perl.
90.9
67.9
66.6
53.9
67.8
73.9
Proprietary
On spatio-temporal tasks (RDCap, DCap, RT- Loc), open source baselines are unable to perform grounded reasoning and default to repeating the same caption for every time interval. Proprictary models perform reasonably well, yet far from the human performance. In sub-tasks of PLM- VideoBench, PLM shows competitive performance compared to proprictary and ppen-sccess baselines. Results for all model scales are in Appendix E.
GPT-4o [6]]
61.2
63.7
20.9
35.7
33.1
51.6
Gemini [63]
57.1
49.9
14.4
27.6
44.0
Gemini 2.0 Hush [63]
58.7
44.8
13.2
30,9
27.6
42.5
Open-aceess
LLaVA-OV:7B [64]
4.7
24.4
Ques2VL-78 [16]
49.2
44.5
4.1
17.6
15.1
35.3
Qve22.5VL-7B [15]
2.5
34.8
IaterVL2 8D [10]
47.7
45.9
1.2
21.5
11.6
35.0
InterVLL2.5-8B [10]
53.7
48.3
5.7
26.1
38.5
PLM-B
46.2
52,8
Table 5:
results. We evaluate PLM against baselines and report breakdowns. We report human performance in the first row.
Tab. 18 in the Appendix reports PLM's 's perfor- mance when trained without any aunan-annotated in-distribution samples. Even in this setting, PLM outperforms Qwen2.5-VL. 3B and Intern VL2.55 4B on PLM- BideoBecc..
Note that the human performance varies based on the nature of the task and evaluation metrics. For example, FGQA human scores are naturally higher than RCap because the task is structured (select the correct option vs. open-ended) and the metric is objective (accuracy vs. LLM-judge accuracy).
5.5 Ablation Studies
Setup. We perform an ablation study to assess the importance of each of our proposed data, both synthetic and human-annotated. We start with PLM 3B after stage 2 training, and finetune on 4M short image and video SFT data mix for the data ablation. We evaluate and report average video benchmark performance across five categories and video hallucination, as well as spatial and temporal tasks, PLM- VideoBench and three image categories image OCR, image captioning and image perception. Full details are in Appendix A.3.
video captioning. short video QA, fine-grained QA,
Discussion. First, we observe that stage 2 synthetic data training boosts model performance across the board. Moreover, adding our PLM-STC data further improves a variety benchmarks, including PLM-STC (+27.4 points), video captioning (+2.44 points), and most importantly, spatial and temporal tasks (+6.8 points). Adding our PLM-FGQA data improves a distinct set of categories for fine- grained activity understanding: PMM-FGQA (+13. 13.1 points), PLM-SGQA (+7.3 points), Fine-grained video tasks 1+3 points), video hallucination tasks (+3.0 points), and spatial and temporal tasks (+2.2 points). Using our Humaan-nnnoated data altogether results in the best performance overall. Further in 7 we show that our human-annotate annotated data improves upon HardQA [35-41], effectively addressing the limitations of synthetic data discussed in $3.2.
33.8M datamix: TextQA 500K, Image QA 2.8M. and Video QA 500K. Each detail be found in Tab, 10.

---

Finally, we conduct an ablation study (Tab. 17 in Appendix). where Stage 2 and 3 are merged into a single training phase using a combined data blend under the Stage 3 setup (36 tiles. 32 frames). Despite nearly doubling the total training FLOOPs, this unified setting yields worse results than the stage pipeline dropping
dioppineddroppin 1.2 points on image benchmarks and 2.0 points on video benchmarks.
PLM VideoBench
Mideo Tunks
Image Pasks
Fine Grined QA
hon Monchmorrk arg.
A Aonchhmrr mrnccmmur ong
michook arz
crge
PLA Sp..-ppohh
PLMSTTC
PL PLM.
Total Averaee
PLM
PLM.SDQA
n org.
Aonchbmark
Video Cap.
Drrom JK
Video QA
Video Hallu.
Spudio& Tomp.
Image OCR
6 6encchoor
fmags Cap
Imags Roc.
46-
PLM
48.5
39.7
34.1
6.6
12.2
24.0
67.5
64.9
50.6
76.0
61.3
63.3
54.1
49.8
35.9
14.7
48.8
29.9
73.2
73.3
56.
1
84.0
65.9
10-
57.9
49.9
36.2
42.1
48.6
32.3
73.9
74.2
62.9
83.8
67.5
65.0
56.7
62.9
43.2
15.2
50.1
30.4
74.1
76.3
58.5
83.7
64.0
61.2
63.6
44.0
42.2
76.3
64.3
8377
26-
Table 6: Ablation. We show the impact of individual data components in PLM training. For this ablation, we use a reduced the SFT datamix consists of 4M open- access image and video data, Results are aggregated validation-set performance over selected benchmarks in cach category of tasks, details in Appendix A.3.
Figure 7: HardQA im-
proves with PLM data.
Related Work
Vision-Language Models. Building the strengths of large language models (LLMs), several vision-
language models (VLMs) have recently been proposed for image understanding [1, 13, 18, 72, 78-82], video understanding [52, 83-89] and joint understanding of both images and videos [10, 16, 64, 90]. These works employ several modeling advancements such as dynamic high resolution inputs [78]. adaptive token compression [87, 91], and multimodal positional embeddings [16].
Open source, open data VLMs. Training data is key component in developing powerful VLMs. Many existing approaches train on proprietary data that is not released to the community [61- 63, 92, 93] or on data generated using proprietary models (e.g., GPT4o) [3], effectively distilling the closed models. Doing so make measuring scientific progress difficult and limits research on how to train VLMs ground-up. Molmo [11] proposes a class of open-data models, however, they are image VLMs trained on relatively small-scale data, limiting their performance as our experiments will show.
VLM Benchmarks. Several benchmarks have been proposed to assess the capabilities of VLMs. Popular image benchmarks cover broad perception and reasoning [29, 33, 68, 69, 71, 72, 94-100] as well as capabilities like image captioning [101-103], documen//diagram understanding [26-28, 65- 67, 70, 104-106], mathematical reasoning [107-109], visual grounding [77, 110] and hallucina- tion [32, 111]. Popular video benchmarks cover video question answering [8, 20, 21, 23, 24, 50- 52, 84, 112-117]. video captioning [47, 59, 74, 1181, and hallucination videos [37, 76]. Many of these video benchmarks remain image-centric - they have questions that can be answered with few frames. Video-centric reasoning benchmarks has been relatively neglected with benchmarks proposed only recently for long video understanding [22, 35, 41, 48, 122-126] and fine-grained, temporal reasoning [36, 38, 39, 75, 127]. We introduce PI. VideoBench- a benchmark suite aimed at the core, video-centric capabilities that current benchmarks neglect, namely fine-grained activity understanding and spatio-temporally grounded reasoning.
7 Conclusion
This work presents Perception Language Model (PLM), fully-reproduciblle vision-language model to transparently tackle visual perception tasks without distillation of private black-box models. We trained PLM using data from existing open-access datasets and synthetic samples generated by our data engine. We identified gaps in detailed video understanding capabilities that cannot be filled with synthetic data. In response, we collected 2.8M human-labels for fine-grained video question answering and apatio-temporeg VideoBench, to evaluate these capabilities. We hope our open dataset, benchmark, and models will foster transparent research in visual perception.
grounded captioning, and created a new benchmark, PLM-
10

---

References
[1]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
[2] Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang. Kunchang Li, Yi Wang, Wenhai Wang. Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang, Limin Wang, Yu Qiao, and Jifeng Dai. Sharegpt-4o: Comprehensive multimodal annotations with go, 2024.
[3] Lin Chen, Jinsong Li. Xiaoyi Dong. Pan Zhang. Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370-387. Springer, 2024.
[4] Farré Miquel, Marafioti Andres, Tunstall Lewis, von Werra Leandro, Conghui He, Cuenca Pedro, and Wolf Thomas. Finevideo: behind the scenes, 2024.
[5] Yuanhan Zhang. Jinming Wu, Wei Li, Bo Li. Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024.
[6] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang. Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang- Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint 241x:2406004325, 2024.
[7] Yuhui Li. Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees, 2024b. URL htips://arxiv. org/abs/2406.16855 2024.
[8] Linjic Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hicrarchical encoder for vidco+ language reperepprsonnttaton pre-training. arXiv preprint 200s 00200, 2020.
[9] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10668-10677, 2020.
[10] Zhe Chen, Weiyun Wang. Yuc Cao, Yangzhou Liu, Zhangwei Gao, Erfci Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05277 2024.
[11] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, Jiasen Lu. Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-But Burcch Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng. Jon Borchardt, Dirk Groeneveld, Jen Dumas, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schocnick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-ar multimodal models. arXiv preprint arXiv:2409.17144 2024.
[12) Daniel Bolya, Po- Yao Huang. Peize Sun, Jang Hyun Cho. Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504. 13181, 2025.
[13] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama herd of models. arXiv preprint arXiv:2407.221 2478.2 2024.
[14] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Speneer Whitehead, Alexander C Berg. Wan-Yen Lo, et al. Segment anything. In Proceedings of the IRFF//VF international conference on computer vision, pages 4015-4026, 2023.
[15] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijic Wang, Jun Tang, al. Qwen2. 5-yl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[16] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: vl: Enhancing language model's perception of the world at any resolution. arXiv preprint arXiv:2409.11 12191, 2024.
[17] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 24185-24198, 2024.
[18] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:200.112597. 2023.
11

---

[19] Brandon Castellano. PySceneDetect.
[20] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405. 2024.
[21] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world vidcos. In Thirry-fifth Conference on Neural Information Processing Systems (NeurlPS), 2021.
[22] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural Information Processing Systems. 36, 2024.
[23] Kunchang Li. Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195 -2226,, 2024.
[24] Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang. Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024.
[25] Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[26] Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Doevqa: A dataset for vqa on dacument images. In 202/ IEEE Winter Conferennee on Applications of Computer Vision (WACV), pages 2199-2208, 2021.
[27] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. Infographicvga. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2582-2591, 2022.
[28] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-L.in Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12)220102, 2024.
[29] xai.
Realworldga benchmark. https:/ http///hugggggac////ee co/datasets/ xxi waaa-oorr//ewwooldaee 2024.
[30] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[31] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering, 2017.
[32] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large vision- langua- models. arXiv preprint arv: 10355, 2023.
[33] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller. Robin Miller, Aubrey Tatarowiez, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. pages 333-3043003 302,3422222 2010.
[34] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019.
[35] Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang. Tong Lu, and Limin Wang. Cg- bench: Clue-grounder grounded question answering benchmark for long video understanding. arXiv preprint arXiv:2412. 12075, 2024.
[36] Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models. arXiv preprint arXiv:2410.2322 2024.
[37] Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu-Gang Jiang. Eventhallusion: Diagnosing event hallucinations video llms. arXiv preprint arXiv:2409. 16597, 2024.
[38] Wenyi Hong, Yean Cheng, Zhuoyi Yang, Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, and Jie Tang. Motionbench: Benchmarking and improving for vision language models. arXiv preprint arXiv:2501 .09555, 2025.
video motion understanding
[39] Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. Temporalbench: Benchmarking multimodal video models. arXiv preprint arXiv:2410.1088 10818, 2024.
temporal understanding for
12

---

[40] Jiyang Gao, Chen Sun. Zhenheng Yang, and Ram Nevatia. Tall: language query. In Proceedings the IEEE international conference on computer vision, pages 5267- 5275, 2017.
Temporal activity localization via
[41] Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang. Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint WrXiv:2406.08033 2024.
[42] Antoine Miech, Dimitri Zhukov, Baptistt Alayrac, Makaaann Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning a text-video embedding by watching hundred million narrated vidco clips. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
[43] Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang. Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullic Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao. Minh Vo, Yuchen Wang. Xindi Wu, Takuma Yagi. Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bemard Ghanem, Vamsi Krishna Ithapu. C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric vidco. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[44]
Kristen Grauman, Andrew Westbury. Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Bymee Zachary Chavis, Joya Chen, Feng Cheng. Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, María Escobar, Cristhian Forigua, Abrham Kahsay Gebreselasie, Sanjay Haresh, Jing Huang. Md Mohaiminul Islam, Suyog Dutt Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J Liang. Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrrsyyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh K. Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song. Shan Su, Zihui Xue, Edward Zhang. Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang. Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mingjing Luo. Zhengyi Luo. Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng. Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram. Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang. Yuchen Wang. Takuma Yagi. Mingfei Yan. Xitong Yang, Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelácz, Gedas Bertasius, David J. Crandall, Dima Damen, Jakob Julian Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C. V. Jawahar, Richard A. Newcombe, Hyun Soo Park, James M. Rehg. Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, and Michael Wray Ego-exoAd: Understanding skilled human activity from first- and third-person perspectives. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19383-19400, 2023.
[45) Yansong Tang, Dajun Wang. Zhenyu Xu, Jingjing Liu, Xiaoyong Wang, Xing Gao, Jinhui Tang, and Dong Wu. Coin: A large-scale scale dataset for comprchensive instructional video analysis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[46] Dimitri Zhukov, Baptiste Alayrac, Chen Sun, Ivan Laptev, Cordelia Schmid, and Josef Sivic. Cross- task weakly supervised learning from instructional videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[47] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[48] Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint D7Xiv:240500881. 2024.
[49] Thong Thanh Nguyen, Zhiyuan Hu, Xiaobao Wu, Cong-Duy T Nguyen, See- Ng, and Anh Tuan Luu. Encoding and controlling global semantics for long-form video question answering. arXiv preprint arXiv:2405. 19723, 2024.
[50] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of dnestion-answetin to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-9786. 2021.
13

---

[51] Zhou Yu. Dejing Xu. Jun Yu. Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127-9134, 2019.
[52] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arX1v:2330.054244 2023.
[53] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmect Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenen- baum. Clevrer: Collision events for video representation and reasoning, arXiv preprint 1rXiv:1910.01442 2019.
[54] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vjjyanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 17XXiv:705.0660 7705.06550 2017.
[55] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller- Freltrr et al. The" something something" vidco database for learning and evaluating visual common sensc. In Proceedings of the IEEE international conference on computer vision, pages 5842-5850, 2017.
[56] Paul Voigtlaender. Soravit Changpinyo, Jordi Pont- Tuset, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with video localized narratives. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2461-2471, 2023.
[57] Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 00714. 14, 2024.
[58] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970, 2015.
[59] Xin Wang. Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4581-4591, 2019.
[60] Soichiro Fujita, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. Soda: Story oriented dense video captioning evaluation framework. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part VI 16, pages 517-531. Springer, 2020.
[61] OpenAT. Gpt-4o system card, 2024.
[62] Gemini Team Google. Gemini: a family of highly capable multimodal models. arXiv:2312. 11805, 2023.
arXiv preprint
[63] Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05520 2024.
[64] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Fong Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.00 23330:2200..03336
2024.
[65] Hanwen Zheng. Sijia Wang, Chris Thomas, and Lifu Huang. Advancing chart question answering with robust chart component recognition. arXiv preprint arXiv:2407.21038, 2024.
[66] Amanpreet Singh. Vivek Natarjan, Meet Shah, Yu Jiang. Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2317-8326. 2019.
[67] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images, 2016.
[68] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmim: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEED/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[69] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A A-okvqa: A benchmark for visual question answering using world knowledge, 2022.
[70] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Secd Sced-bench: Benchmarking multimodal lims with generative comprehension. arXiv preprint arXiv:2307. 16/25, 2023.
[71] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Whiu Ma. and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148 188-166,200 2025.
14

---

[72] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu. Sai Charitha Akula, Jihan Yang. Shusheng Yang. Adithya Iyer. Xichen Pan, Austin Wang. Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: I: A fully open, vision-centric exploration multimodal Ilms, arXiv preprint arXiv:2406. 16860, 2024.
[73] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635 651, 2023.
[74] Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634, 2024.
[75] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang. Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand vidcos? arXiv preprint arXiv:2403.0044 00476, 2024.
[76] Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models, arXiv preprint arXiv:2406. 16338, 2024.
[77] Jierun Chen, Fangyun Wei, Jinjing Zhao, Sizhe Song, Bohuai Wu, Zhuoxuan Peng, S-H Gary Chan, and Hongyang Zhang. Revisiting referring expression comprehension evaluation in the era of large multimodal models, arXiv preprint arXiv:2406. 16866, 2024.
[78] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[79] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Zhang, and Fei Huang. mpplug-ow2:: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13040- 13051, 2024.
[80] Jean-Bane Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson. Karel Lenc. Arthur Mensch. Katie Millican. Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadch, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. arXiv preprint wrXiv:2204.14198 2022.
[81] Ji Lin, Hongxu Yin. Wei Ping, Pavio Molchanov, Mohammad Shocybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 26689-26699, 2024.
[82] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma. Chengyue Wu, Bingxuan Wang, al. Deepseck-v12: Mixture-of-experts vision- laniuug models for advanced multimodal understanding. arXiv preprint arXiv:2412. 10300 2024.
[83] KunChang Li, Yinan He, Yi Wang, Yizhuo Li. Wenhai Wang. Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Vidcochat: Chat-centric vidco understanding. arXiv preprint MXXiv:23005063355 2023.
[84] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding, arXiv preprint arXiv:2406.0944 094188 2024.
[85] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.0012 2023.
[86] Jiajun Liu, Yihing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint ar0iv:2408. 15542, 2024.
[87] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporaa adaptive compression for long video-language understanding. arXiv preprint 4rXiv:2410.174344 2024.
[88] Yuctian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvim: Efficient long video understanding via large language models. In European Conference on Computer Vision, pages 453-470. Springer, 2025.
[89] Xiaohan Wang. Yuhui Zhang, OrT Zohar, and Serena Yeung-Levy. Videoagent: Long-form vidco understanding with large language model as agent. arXiv preprint arXiv:2403. 201.22403.10117, 2024.
[90] Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, et al. Anymal: An efficient and scalable any any-modality augmented language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1314-1332, 2024.
15

---

[91]
Rohan Choudhury. Guanglei Zhu, Sihan Liu, Koichiro Ninuma. Kris M Kitani, and László Jeni. Don't look twice: Faster video transformers with run-length tokenization. arXiv preprint arXiv:2411.05222.
2024.
[92] OpenAI. Gpt-4v(ision) system card, 2023.
[93]
Anthropie. The claude 3 model family: Opus, sonnet, haiku, 2024.
[94] Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui Hc, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2024.
[95] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[96] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin. Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.022 02490, 2023.
[97] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiagi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint 17XXv:2403.20333 2024.
[98] Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in wild with human preferences. arXiv preprint arXiv:2406. 11069, 2024.
[99] Dongfu Jiang, Xuan He, Huaye Zeng. Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405. 014483, 2024.
[100] Fei Wang, Xingyu Fu, James Y Huang. Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, al. Muirhench: comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406. 094/1, 2024.
[101] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[102] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision. pages 8948-8957, 2019.
[103] Peter Young, Alice Lai, Micah Hodash, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational I Linguistics, 2014.
[104] Zirui Wang, Mengzhou Xia, Luxi Hc, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding multimodal llms. arXiv preprint arXiv:2406. 2200.18521, 2024.
[105] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6720-6731, 2019.
[106] Pan Lu. Swaroop Mishra. Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song- Zhu, Oyvind Tafford, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Curran Associates,
Advances in Neural Information Processing Systems, volume 35, pages 2507-2521.
Inc., 2022.
[107] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Rei Chang. Yu Qiao, et al. Mathverse: Does your multi-modal I1m truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186, 2025.
[108] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 2023.
[109] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Mcasuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402.1 14804, 2024.
[110] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32 73, 2017.
16

---

[111] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375-14385, 2024.
[112] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758-2766, 2017.
[113] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara I. Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXivv:809. 02 696,, 2018.
[114] Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video- bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16100 2023.
[115] Jianrui Zhang. Mu Cai, and Yong Jae Lee. Vinoground: Serutinizing Imms over dense temporal reasoning with short videos. arXiv preprint arXiv:2410.0276. 2024.
[116] Xinyu Fang. Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li. Dabua Lin, and Kai Chen. Mmbench-video: A forg-form multi-shot benchmark for holistic video understanding, arXiv preprint arXiv:2406. 14515, 2024.
[117] Daniel Cores. Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano. Tvbench: Redesigning video-languge evaluation. arXiv preprint a:Xiv:2410.0772, 2024.
[118] Jun Xu, Tao Mci, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging vidco and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288 0296,2996,222 2016.
[119] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190-200, 2011.
[120] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events videos. Proceedings of the IEEE international conference on computer vision, pages 706-715, 2017.
[121] Wenhao Chai, Enxin Song, Yilun Du. Chenlin Meng. Vashisht Madhavan, Omer Bar-Tal. Jeng-Neny Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.0305), 2024.
[122] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through conference on computer vision and pattern recognition, pages 4631-4640 2016.
answering. In Proceedings of the IEEE
[123] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideohench: benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828- 28857, 2025.
[124] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 18221-18232. 2024.
[125] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi- long video understanding. arXiv preprint arXiv:2406.0422
2024.
[126] Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen -estruch,, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. arXiv preprint 2412 103300 2024.
[127] Mohammadreza Salchi, Jae Sung Park, Tanush Yadav, Aditya Kusupati, Ranjay Krishna, Yejin Choi, Hannaneh Hajishirzi, and Ali Farhadi. Actionatlas: A videoqa benchmark for domain-specialized action recognition. arXiv preprint arXiv:2410.05774, 2024.
[128] Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05100 2017.
[129] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer. et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11 110711 2023.
[130] Tri Dao. Flashattention-2 2: Faster attention with better parallelism and work partitioning. arXiv preprint mrXii:2307.0861, 2023.
17

---

[131] Adam Paszke, Sam Gross, Soumith Chintala. Gregory Chanan, Edward Yang, Zachary Vitioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Alan Lerer. Automatic differentiation in pytorch, 2017.
[132]
Montalvo Pablo and Wightman Ross. Pdf association dataset (pdfa), 2024.
[133]
Montalvo Pablo and Wightman Ross. Industry documents library (idl), 2024.
[134] Lei Li. Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.00 00231, 2024.
[135] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-saale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430 8439, 2019.
[136] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Mattco Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scalc. IJCV, 2020.
[137] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in neural information processing systems, 34:23634-236111, 2021.
[138] Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 14871- 14881, 2021.
[139] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Holly- wood in homes: Crowdsourcing data collection for activity understanding, In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 14, 2016, Proceedings, Part 1 14, pages 510-526. Springer, 2016.
[140] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812, 2017.
[141] Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang. Ila Kulikov, Kyunghyun Cho, Yuandong Tian, Jason Weston, al. Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions. arXiv preprint arXiv:2502. 13124, 2025.
[142] Kushal Kafle, Scott Cohen, Brian Price, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018.
[143] Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In The IEEE Winter Conference on Applications of Computer Vision (WACV), March 2020.
[144] Shuaichen Chang, David Palzer. Jialin Li, Eric Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering on choropleth maps, 2022.
[145] Anand Mishra. Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 947-952, 2019.
[146] Jordi Pont-Tusee Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives, 2020.
[147] Samira Fhrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning, 2018.
[148] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2611 1-224.4. Curran Associates, Inc., 2020.
[149] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zätnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016.
[150] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021.
[151] Mehran Kazemi. Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning, 2023.
18

---

[152] Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi. Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. Robut: A systematic study of table qa robustness against anman-annotated adversarial perturbations. In Anna Rogers, Jordan Boyd-Grnd boyd-Graber, and Naoaki Okazaki, editors. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6064 6081, Toronto, Canada, July 2023, Association for Computational Linguistics.
[153] Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset, 2024.
[154] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w:
Grounded question answering in images. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[155] Manoj Acharya, Kushal Kafle, and Christopher Kanan, Tallyqa: Answering complex counting questions.
In AAAI, 2019.
[156] Jonas Belouadi, Anne Lauscher, and Steffen Eger. Automatikz: Text-guided synthesis of scientific vector graphics with tikz, 2024.
[157] Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering.
In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[158] Jason Obeid and Enamul Hoque. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In Brian Davis, Yvette Graham, John Kelleher, and Yaji Sripada, editors, Proceedings of the 13th International Conference on Natural Language Generation, pages 138-149 Dublin, Ireland, December 2020. Association for Computational Linguisties.
[159] Benny J. Tang, Angie Boggust, and Arvind Satyanarayan. Vistext: A benchmark for semantically rich chart captioning. In The Annual Meeting of the Association for Computational Linguistics (ACL), 2023,
[160] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, lana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. Finqa: datasct of numerical reasoning over financial data. In Wen-tau Yih, editors. Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 3697-3711, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Moens, Xuanjing Huang. Lucia Specia, and Scott
[161] Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluis Gomez, Marçal Rusiñol, Jawahar, Emnest Valveny, and Dimosthenis Karatzas. Scene text visual question answering. In 2019 IEEF/CVF International Conference on Computer Vision (ICCV), pages 4290-4300, 2019.
[162] Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benehmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3277-3287, Online, August 2021, Association for Computational Linguistics.
[163] Chris Wendler. Renderedtext, 2024.
[164] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Patten Recognition (CVPR), 2019.
[165] Urs-Viktor Marti and H. Bunke. The iam-database; An english sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition, 5:39 -6. 11 2002.
[166] Pan Lu, Liang Qiu, Kai-Wei Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tammay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023.
[167] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension, 2020.
[168] Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Sereen2words: Automatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, UIST '21, page 498-510, New York, NY, USA, 2021. Association for Computing Machinery.
[169] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning, 2023.
[170] Aniruddha Kembhavi, Minjoon Sco, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannanch Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5376-5384, 2017.
[171] Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on document images. In AAAI, 2021.
19

---

[172] Jason Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific Data, 5:180251, 11 2018.
[173] Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, GannG Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1094-1110, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[174] Pan Lu. Ran Gong, Shibiao Jiang. Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Zhu. Interpretable geometry problem solving with formal language and symbolic reasoning. In The Joint Conference of the 59th Annual Meeting the A Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACI-IJCNLP 2021), 2021.
Inter-gps:
[175] Diagram image to text dataset, 2023.
[176] Bo Li. Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. 2023.
[177] Yilun Zhao, Yunxiang Li. Chenying Li, and Rui Zhang. Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association Dublin, Ireland, May 2022.
for Computational Linguistics (Volume I: Long Papers). pages Association for Computational Linguistics.
[178] Alane Suhr, Stephanic Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. In Anna Korhonen, David Traum, and Lluís Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418-6428, Florence, Italy, July 2019. Association for Computational Linguistics.
[179] Harsh Jhamtani et al. Learning to describe differences between pairs of similar images. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun 'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural I Language Processing, pages 4024-4034, Brussels, Belgium, October- November 2018. Association for Computational Linguistics.
[180] Haoping Bai. Shancong Mou, Tatiana Likhomanenko, Ramazan Gokberk Cinbis, Oncel Tuzel. Ping Huang, Jiulong Shan, Jiamjun Shi, and Meng Cao. Vision datasets: A benchmark for vision-based industrial inspection. arXiv preprint ar Xiiv:2366.07890 2023.
[181] Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. Imagine this! scripts to compositions to videos. In Proceedings of the European conference on computer vision (ECCV), pages 598 613, 2018.
[182] Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image retrieval from contextual descriptions. arXiv preprint arXiv:2203. 15867, 2022.
[183] Phillip Isola, Joseph Lim, and Edward II Adelson. Discovering states and transformations in image collections. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 1391-1391, 2015.
[184] Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao. Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. Proceedings of the IEEF/CVF conference on computer vision and pattern recogmition, pages 16495-16504, 2022.
[185] Maxwell Forbes. Christine Kaeser- Piyush Sharma, and Serge Belongie. Neural naturalist: Generat- ing fine-grained image comparisons. arXiv preprint arXiv:1909.044: 2019.
[186] Hareesh Ravi, Kushal Kafle, Scott Cohen, Jonathan Brandt, and Mubbasir Kapadia. Acsop: Abstract encoding of stories, objects, and pictures. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2052-2063, 2021.
[187] Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler- Cinbbs, Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. arXiv preprint 1r09i 2018.
[188] Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4624-4633 2019.
[189] Rumeysa Bodur, Erhan Gundogdu, Binod Bhattarai, Tae-Kyun Kim, Michael Donoser, and Loris Bazzani. iedit: Localised text- text-guided image editing with weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7426-7435, 2024.
[190] Panupong Pasupat and Percy Liang. Compositional semantic parsing semi-structured tables. In Chengging Zong and Michael Strube, editors, Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 1470- 14800 Beijing, China, July 2015. Association for Computational Linguistics.
20

---

[191] Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji. Zhonggin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.0160) 2022.
[192] Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jacmin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont- Pont-Tuset, Garrett Tanzer, et al. Docci: Descriptions of connected and contrasting images. In European Conference Computer Vision, pages 291-309. Springer, 2024.
[193] Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson. Vasu Sharma, and Adriana Romero- Soriano. A picture worth more than 77 text tokens: Evaluating clip-style models dense captions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 26700- 26709, 2024.
[194] Hu Xu, Pao Huang, Xiaoging Ellen Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, et al. Altogether: Image captioning via re-aligning alt-text. arXiv preprint 1xXi::2410.172555 2024.
[195] Drew Hudson and Christopher D Manning. Gqa: A new dataset for real- real-world visual reasoning and compositional question answering. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[196] Bryan A Plummer, Liwei Wang, Chris M Cervantes. Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentenee models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649. 2015.
[197] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[198] Xiyao Wang. Yuhang Zhou, Xiaoyu Liu, Hongjin Lu. Yuancheng Xu, Feihong He, Jachong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, Huaxiu Yao, and Furong Huang. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences, 2024.
[199] Abhay Zala. Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video- retrieval and step-captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23056-23066 2023.
[200] Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 32(11))8238-824 2021.
[201] Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots. https: httpa://huggnngfacce ca/ ht////aaatttt/uugggaa wawegggingaaee///oo...
2023.
[202] Aida Amini, Saadia Gabriel, Peter Lin, Rik Soncel-Kedziorrkk Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation based formalisms. arXiv preprint arXiv:1905. 10000/113199 13319, 2019.
[203] Chunting Zhou, Pengfei Liu, Puxin Xu. Srinivasan Iyer, Jiso Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less more for alignment. Advances in Neural Information Processing Systems, 36:55006-55021, 2023.
[204] Karl Cobbe. Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton. Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 10,1:1101,1116 14168, 2021.
[205] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
[206] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly; Introducing the world's first truly open instruction instruction-tuned 1lm, 2023.
[207] Zhangchen Xu, Fengging Jiang, Luyao Niu. Yuntian Deng, Radha Poovendran. Yejin Choi, and Bill Yuchen Lin. Magpie: Aligument data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint 240i::20000084664 2024.
[208] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2015.
[209] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024.
21

---

[210] Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang. Lin Chen. Yuan Liu, Xiaoyi Dong, Yuhang Zang. Pan Zhang, Jiaqi Wang. et al. VImevalkit: An open-source toolkit for evaluating large multi- modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11198-11201, 11101,112220 2024.
[211] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303, 05499, 2023.
[212] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Universal instance perception object discovery and retrieval. In CVPR, 2023.
[213] Peng Wang, Shijie Wang. Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang. and Chang 7hou. One-peace: Exploring one general representation model toward unlimited modalities. arXiv preprint wxXi:2305.11172 2023.
[214] Jang Hyun Cho. Boris Ivanovic, Yulong Cao, Edward Schmerling, Yue Wang, Xinshuo Weng, Boyi Li, Yurong You, Philipp Krachenbuchl, Yan Wang, and Marco Pavone. Language-image models with 3d understanding. In The Thirteenth International Conference on Learning Representations. 2025.
[215] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision-ECCV 2016; 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
[216] Yale Song, Eugene Byme, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Information Processing Systems, volume 36. pages 38863-38886. Curran Associates, Inc., 2023.
[217] Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagarajan, Huiyu Wang, and Lorenzo Torresani. -t--step: Aligning instructional articles with how-to vidcos. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[218] Effrosyni Mavroudi, Triantafyllos Afouras, and Lorenzo Torresani. Learning to ground instructional articles in videos through narrations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15201-15213, October 2023.
[219] Yi Wang. Kunchang Li. Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang. Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang. et al. Internvide22: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403. 15377, 2024.
[220] Hyolim Kang, Jinwoo Kim, Tachyun Kim, and Seon Joo Kim. Uboco: Unsupervised boundary contrastive lcarning for generic event boundary detection. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20041-20050, 2021.
[221] Zexing Du Xue Wang, Guoqing Zhou, and Qing Wang. Fast and unsupervised action boundary detection for action segmentation. In 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3313-3322, 2022.
[222] PySceneDetect: Video Cut Detection and Analysis Tool, pyscenedetect.
[223] J. S. Chung and A. Zisserman. Out of time: automated lip sync in the wild. In Workshop on Multi-view Lip-reading, ACCV, 2016.
[224] 7i-Yi Dou, Xitong Yang, Tushar Nagarajan, Huiyu Wang, Jing Huang, Nanyun Peng, Kris Kitani, and Fu-Jen Chu. Unlocking exocentric video-language data for egocentric video representation learning. ArXiv, abs/2408.03367, 2024.
[225] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact internet scale. In CVPR, 2020.
[226] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling spen-vocayulary object detection. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023.
[227] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
[228] Andrew Jaegle, Sebastian Borgeaud, Jean- nan--Bptiste Alayrac Carl Doersch, Catalin lonescu, David Ding. Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. ICLR, 2022.
22

---

[229] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Pattern Recoggition, 2016.
[230] Sergi Caelles, Jordi Pont-Tuset, Federico Perazzi, Alberto Montes, Kevis-Kokise Maninis, and Luc Van Gool. The 2019 davis challenge vos: Unsupervised multi-object segmentation. arXiv:1905.00737, 2019.
[231] Yan Yan, Chenliang Xu, Dawen Cai, and Jason J Corso. Weakly supervised actor-action segmentation via robust multi-task ranking. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1298-1307, 2017.
[232] Ujjal Kr Dutta, Mehrtash Harandi, and Chellu Chandra Sekhar. Unsupervised deep metric learning via orthogonality based probabilistic loss. IEEE Tronsactions Artificial Intelligence, 1(1):74-84, 2020.
[233] I Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason Corso, and Marcus Rohrbach. Grounded video description. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6378-6587, 2019.
[234] Seonguk Seo, Joon- Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with large-scale benchmark. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XV 16, pages 208 223. Springer, 2020.
[235] Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 32(12):82388824 2021.
[236] Henghui Ding, Chang Liu, Shuting Hc, Xudong Jiang, and Chen Change Loy. Mevis: large-scale bench- mark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2694-2703, 2023.
[237] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound, In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 16375-16387 2022.
23

---

Appendix
Table of Contents
A PLM Training Details
25
A.1 PLM Training Setting
25
A.2 PLM Training Datamix
26
Ablation Experiment Details
26
B Synthetic Scaling Experiments
29
C VLM Benchmark Details
31
C.I. Image Benchmarks
31
C.2 Video Benchmarks...
31
PLM-VideoBench
31
C.4 Evaluation Protocols
32
D Baseline Implementation Details
33
E
Additional Results
34
E.1
Additional PLM- VideoBench Results
34
E,2
Comparison with LLaMA-3V
.et.............................
34
Image Captioning
34
E.4 Image Grounding
................................
34
E.5
Long Video Understanding
34
E.6
Additional Ablations
35
F PLM-FGQA: Fine-grained QA
37
F11 Annotation process: Data Engine
37
FGQA PeMe Construction
G PLM-STC Details
44
G.1 Annolation Process
44
G.2 PLM-STC Benchmark
45
H Smart Glasses Data
47
H.l Data collection and annotation
47
SGQA Benchmark
47
I Synthetic Data Engine
48
J Qualitative Results
52
K Limitations and Future Work
56
L Broader Impact
56
24

---

A PLM Training Details
Image- Synthetic
Video Synthetic
Video Human
Open Source
Documments
Exocentric Videes
Finee- Finn-grainndd
Images
L pinitteee (eTe,t0 S0o
L Cappiin Cejeessees
47.8M [0un]]
Egocentric Videos
Coetering of Thmeest 1900
Orhire 200g)
18A4M
(Oung)
Litoo 24 3344
Spatio-Tem
L Connee S04
Vidaac
6.52M
Natural Images
L (SVi) Chteeooss Open 20/64
Captonn Ageso UaO
Ruskcap 3006
Ditassu floa 1754)
Total 69.1M PLM Image and Video Data
6.52M Existing Data
Perception Language Model (PLM)
Image QA
Q:
& shown imsgge?
What location dees this mopp rspresent?
PLLMARD PLM MMM- MuRt mennne
Q: Spot the diffore noe betwoon images.
As werees se isic scere with monumen: featur rg man
Image
Ac the map is Kanaas interationall icnal Arpprt IKK,, wiic a locabed Japan
Aoe co or of the pole on the digge right changes from black light
Video QA
Qo What the persons doingg
Video Cap
rifffly caption iido.
Q: Now dowe is paeson pour the mug?
The video shows a person makk ng wincetts sandwich by mashing the avcceeo
A: Pengin le woeringg ting in
Ac The penion movess the tespor up and etown those timee while pituring
left hand and cursing the
spreading
muug.
PLM RTL
Qu Locarize event
PIM-RCap
Descrize the high lightod bsjcc in video betweon
Oo Provide diense caption for region
connert dlamiing: forr the
shacled rugion??
A: 10, 12 jid in playtng hide and
A person focing towards
twek and hetwe behind teee (S%
reappeses
Figure 8: The figure provides an overview the datasets used in the paper. PLM trained with 47.8M synthetic image and 18.4M synthetic video, and 2.9M human-labeled video samples. Our data enables PLM to perform a variety of tasks, including standard tasks like Image, Multi-image, and Video QA, as well as new video tasks such Fine-grained QA (FGQA), Region Temporal Localization (RTL,c)), Region Captioning (RCap), and Region Detailed Captioning (RDCap).
In this section, we describe the training details of PLM. In $A.1 we describe exact details of training setting such as hypee-paaameters and implementation details. In $A.2 we describe our datamix for both synthetically generated and human- annotated parts.
A.1 PLM Training Setting
For all three stages, we use AdamW optimizer [128] with weight decay of 0.05 and use FSDP [129] with FlashAttentio22 [130] for overall implementation based on Py PyTorch [131].
Stage training. In stage 1, we use subset of SA-1B [14] paired with detailed captions generated by our data engine ($3.1), ). We use total 1M samples to train PLM with next token prediction loss, with vision encoder and LLM parameters frozen. This stage is commonly known as warm-up stage. We use learning rate 1 x 10 for all model scale with global batch size of 512 and 448 x 448 resolution. We use the Perception Encoder [12] L/14 variant for the 1B and 3B PLM models, and the G/14 variant for the 8B PLM model.
Stage training. In Stage 2. we train on total of 72.5M samples. Of these, 66M consist of images and videos with synthetically generated annotations produced by our data engine. The remaining 6.5M samples are subset of annotttdd images and videos from open-source datasets, which are included in our final datamix described in $A.2. We train with global batch size of 2048, learning rate of 10- weight decay of 0.05 for the full set of parameters (vision encoder, projector, and LLM). For both image and video input, we use 448 x 448 resolution for each tile/frame, which effectively generate 1024 vision tokens. We apply 2 x spatial average pooling to reduce this to 256. We use dynamic tiling with thumbnail to support any resolution and aspect ratio, similar to prior work [78], and uniform sampling of video frames after preprocessing the videos to fps. We set the maximum number of tiles/frames to be 16, which results in maximum of (16 + 1) x 256 = 4352 and 16 x 256- = 4096 vision tokens respectively for images and videos. We train the model with a sequence length of 6144 allowing maximum of 2048 tokens for the text modality.
Stage 3 training. In stage 3, we use total of 19.1M high-quality datamix spanning over multiple image, video, and text modalities. We describe this datamix in A.2. In this stage, we use global batch size of 1024, learning rate of x 10-5 for 8B and x 10-5 for 1B and 3B PI.M models. We train the full set of parameters for all scales. Similar to stage 2, we adapt dynamic tiling and uniform
25

---

frame sampling for up to 36 tiles for image and 32 frames for video, with 2 x 2 spatial average pooling, which generates (36 + 1) x 256 9472 vision tokens for image and 32 256 8192 vision tokens for video. For all modalities, we use 11264 maximum training sequence length.
Training cost. The full training process for the PLM-3B model, including all three stages, takes approximately 5.5 days on 128 H100 GPUs. The PLM-1B model requires less training time, while the PI.M-8B model takes slightly longer. The detailed breakdown is listed the Table 7 Further, at inference time, all models can run efficiently on single GPU: the 1B model on 24GB GPU, the 3B model on a 40GB GPU, and the 8B model on an 80GB GPU.
Model
Stage
Stage 2
Stage
PLM-IB (PE L/14)
8 GPUs & hours
128 GPUs & 1.5 Days
128 GPUs & 2.0 Days
PLM-3B (PE L/14)
GPUs & hours
128 GPUs & 3.0 Days
128 GPUs & 2.5 Days
PLM-8B (PE G/14)
8 GPUs & hours
256 GPUs & 3.0 Days
256 GPUs & 2.5 Days
Table 7: Training efficiency of PLM models across scales.
A.2 PLM Training Datamix
Table 10 presents the full data mix used across all training stages apart from our manually collected data in $4. This contains annotations from existing public datasets as well as synthetically generated data (see $3). We filter and include wide variety of existing datasets spanning across images (captioning, QA, grounding). videos (captioning, QA, temporal localization, region captioning and dense captioning) and text-only datasets to preserve the text-instruction following capabilities of our model. Most importantly, we filter out every dataset that contains annotations generated by proprictary models. Table 8 and Table 9 shows the exact number of samples for each datasets in Stage 2 and Stage 3 respectively. Marjory of the data in stage 2 are synthetic, with a focus on captioning samples, since they carry the dense information about the image or video. In stage 3, we have one third of the data, mostly focusing on human annotated samples, covering a large variety of tasks.
A.3 Ablation Experiment Details
We provide additional details about the ablation experiment in $5.5. We report benchmark average scores across categories, along with the average across all of them. We select representative set of benchmarks from the full set of image and video benchmarks in $5.2 and $5.3 that report comparable scores so the average results are meaningful. For Video captioning we select Dream 1K and report the LLM-judge score with Llama3.3 70B as judge. for Short Video QA, and Finegrained QA, we select benchmarks that report MCQ accuracy (and exclude open-ended QA). For Hallucination, we include both benchmarks. For Spatial and Temporal tasks, select BLINK, CVBench, VSR, and Charades- -Trra For Image Perception, we choose SEED, MMMU, VQAv2, OK-VQA- and VizWiz. We train the ablation setup of SFT with the exactly matching hyperparameters as final run; only difference is the size of the SFT datamix.
26

---

Dataset
Num Samples
Type
Dataset
Num Samples
Type
Image Synthetic
Image Synthetic
PDFAcc (QA) [132]
12M
QA
PDFAcc (QA) [132]
2M
QA
PDFAce (Cap) [132]
12M
Cap.
ArxivCap [134]
1.5M
Cap./QA
UCSF [133]
6M
QA
SAIB [14]
800K
Cap.
ArxivCap [134]
1.8M
Cap./QA
Object365 [135]
300K
Cap.
SAIB [14]
10M
Cap.
OpenImages [136]
300K
Cap.
Object365 [135]
3.5M
Cap.
DecVQA [26]
100K
QA
OpenImages [136]
1.8M
Cap.
Infographic [27]
50K
QA
DocVQA [26]
50K
QA
PixmoCap [11]
500K
Cap
Infographic [27]
20K
QA
Video Synthetic
PixmoCap [11]
600K
Cap
YT-IB (QA) [137]
300K
MCQA
Video Synthetic
Ego4D (Cap.) [43]
180K
Cap.
YT-IB (Cap.) [137]
14M
Cap.
Ego4D (QA) [43]
700K
QA
YT-IB (QA) [137]
3M
MCQA
Spoken Moments [138]
449K
Cap.
Ego4D (Cap.) [43]
180K
Cap.
Charades [139]
8K
Cap.
Ego4D (QA) [43]
700K
QA
Kinetics710 [54]
40K
Cap.
Spoken Moments [138]
449K
Cap.
DiDeMo [140]
7.5K
Cap.
Charades [139]
8K
Cap.
Text Synthetic
Kineties [54]
40K
Cap.
NaturalReasoning [141]
IM
QA
DiDeMo [140]
7.5K
Cap.
Human Annotated
Text Symthetic
Image QA [10]
2.8M
QA
NaturalReasoning [141]
1M
QA
Image Cap [10]
36K
QA
Human Annotated
Image Gmd. [10]
1.4M
QA
Image QA [10]
2.8M
QA
Image Misc. [10]
1.4M
QA
Video QA [10]
570K
QA
Video QA [10]
570K
QA
Video TL [10]
16K
Temp. Loc.
Video Cap. [10]
315K
QA
Video Dense Cap. [10]
10K
Dense Cap.
Video TI. [10]
16K
TI.
Text QA [10]
2M
Mix
Video Dense Cap. [10]
10K
DCap.
Total
72.5M
Video Region Captioning [10]
15K
Cap.
Text QA [10]
1.5M
Mix
Table 8: PLM Stage training data mix.
Human Annotated (Our)
PLM FGQA
2.4M
QA
PLM STC
476K
Cap./TL
Total
19.1M
Table 9: PLM Stage training data mix.
27

---

Truae OA
Gueedday
Villo Twey on
Dullet
Sez
Deard
Sie
Dukect
Siec
Daaed
Saa
Plagj (145g
155030
NBCTQA115 150
3850
Madde Stnrie [1M]
286332
Churade (15g
7566
(190)
(140)
oCtEVQA [145]
Kanzatoe [196]
167846
100428
Faaaraad [193]
22341
VGI (ive)
213925
855577
Total
15629
Tbaan Mores (144)
9733
MIT Acam Columnane (183)
1001
Tond
CLBVV 140g
Jocntjk (X])
70000
WoKRA (114]
Works [183)
5135
Dusaer
Size
HC- STVG [188)
10131
Goveaeese [131]
31951
11102
Rocas QA Ilg.. Coboxcod (187)
AE40P
6415
$573
DxvVQA Dag
Total
Weldiige (1450
10000
HS8 [199]
1455
BUFFAx (QM [132]
Patfice 11335
53536670
123385/0
Valeo Dinna
1M00/0 [138]
Sober (WTO
100039
DecVQA [X
60562
1859200)
Actritykaa [2])
Nads
CreagA (157)
46387
Toa [14]
69120
Oteaches (135)
Opentnupe [130)
5454356
Chanciaas
Vigav2 (1))
50313
WTQ (19)
34195
25555
581591
Vadeo Synth
Hoat 0dc (191)
76492
2218
chevers wenntag [4/]]
8835
Video
Restondfior [163]
STVQA [1:-1
SAVVEN [154]
(152)
51415
10884
Eavoraay ( (17))
Total
2796545
1833
Niar QA (MD0)
Nos (30)
Size
$4.14
Eac4t <Chp (ucc)
Churaka (109g
Kincti-710 [-4]
7266
7915
183223
CKVVV goo)
11735
NaarRAA (-1)
235.0
V IC (1:7)
(QA) (177)
141529933
CheuQiA (157)
Wivee
Toualh
0009
2157
BOCEE [195)
1999
14505
CI EVSEE pmtt)
Sax2 50
Kreelochi
(Qua (30)
40390
(35.0
Tetal
Datacck
Rober (SDA)
12363
NLEU 10A0
15381
Rusbbo [20g]
9433
fragee
VADA 120
0943
513
Deauei
Skxc
STAR
Monaiio (19]]
40300
35397
LIMA (X01)
(eccratic [364]
7433
HOCC
12415
Tiaah (175)
4565
Oud (101]
414115
Soaal
567710
PLAN (10))
(58883)
G2S dagan Lmage (175)
2005
1361526
Vikeo
Magic (201)
Migii Pro (MT)) (207)
Mariilimm (177)
14331
empireede ESN
Total
2106359
BUWVN (Madi-inagei [164]
SpeTwEkiT (17)
3008:
Chandlo
Autiing Not (mplisea) (38)
33975
31583
Siaal
353218
Table 10: PLM training datamix. Our mix includes synthetic and manually annotated data across combination of image data (QA, captioning, OCR, Visual grounding). video data (captioning, grounded captioning, dense captioning, temporal localization) and text-only data, Importantly, all data is publicly accessible, and not generated by proprictary models.
28

---

B Synthetic Scaling Experiments
In this section we provide additional results to the synthetic scaling experiments in $3.2. We report aggregate benchmark accuracies across three categories Video QA, OCR QA and Image QA- by selecting representative benchmarks from each category. For VideoQA, these are STAR [21], EgoSchema [22]. MVBench [23], VideoMME [20] and PerceptionTest [24]; For OCR QA, these are ChartQA [25], Doc VQA [26], InfographicsQA [27], TextVQA [34] and OCRBench [28]; and for Natural Image QA, these are RealworldQA [29]. OKVQA [30], VQAv2 [31], and VizWiz [33].
Video QA Err. - 18.0e+03 x GFLOPs
OCR QA Err. - (6.6e+03 x CFLOPs)
Natural QA Ert. = 02. x CFLOPs]
No Syth, (16)
50-
48
No Syth. (1B)
Syth. (38)
No Syth. (1B)
Avg. Error
45
42-
Syth, (9B)
40
Syth (3B)
(3th
38
(2.5c+03.0.15
PE+L
Syth (80)
PE-L [6.5e-03,0020
No Syth. (8B)
(6.3e+02-00.1
32
1012
10 CFLOPs
1012
Power Fit- Power Law
Figure 9: Scaling with encoder size. Scaling trends of PE-G vs. PE-L vision encoders. Larger encoders scale better in Video QA tasks while similar scaling in OCR and Natural QA is seen.
Scaling with encoder size. After investigating the impact of the LLM decoder in Fig. 2, we examine the impact of increasing the vision encoder size from 300M (PE Large) to 2B (PE Giant) for each language model scale next. In Fig. 9, we overlay the new power-law with the 2B vision encoder (black dashed) line onto the 300M (red dashed) line. Notably, we find that the larger vision encoder (300M- + 2B) leads to greater scaling trend on video QA benchmarks. Quantitatively, the power law fit has improved from -0.15 to -0.19. The two lines intersect around 8B scale with PE-G, proving that 8B and larger PLM will benefit more with larger vision encoder. We use PE-L for 1B and 3B LLM scale and PE-G for 8B scale by default.
Video QA
OCR QA Err. = (6.8er04 x GFLOPs)
Natural QA Err. (9.7e+ x GFLOPs)
Err. = x3.0e+03 GFLOPS) 0015
50
45
Era Fram.
Error
40
B 044
8 Tiles
Tilles
35
B Fram.
(1.0e 0.15)
Fram. (177-03, 0.13)
Tiles
Tiles
30
Tiles
Tiles
+18
18 5G
25
!
10
10
1012
10ts
1012
10ah
0F-J0T Power Law Fit
16F/GT Power Law Fit
CFLOPs
Figure 10: Scaling with input size. Scaling trends of training with 16 tiles/frames vs. tiles/frames. Higher input size scales better Video QA and OCR QA tasks while similar trend seen for Natural QA.
Scaling with input size. In Fig. 10, show the impact of increasing the input size to VLM through higher image resolution and more video frames. In this setting, each scale of PI.M trains with dynamic tiling for image input and uni sampling for video input with maximum 8 or 16 tiles/frames per sample. In each plot, the average error of PLM trained with 16 tiles/frames are plotted. All models use 2 x 2 spatial average pooling before input to LLM, and each tile/frame has 448 x 448 resolution. Similar to Fig. 2, we show power law fit with a black dashed line, and compare to tiles/frames training denoted with red dashed line. Notably, we find out that on Video QA and OCR QA benchmarks, PLM shows better scalability with training with higher input size. This means with the same FLOP counts at 10-3, training with 16 frames makes 2.0 points of metric error lower than 8 frumes counterparl (32.2 vs 30.2). Similar trends are observed with OCR QA going from 8 tiles max. to 16 tiles max. Notably, higher resolution did not make difference for Natural QA tasks. We chose the 16 max- tiles and frames to be our final training setting for stage 2 PLM.
29

---

VideoMME (710.28, -0.9)
MVBench
EgoSchema (97167.63, -0.26)
(1729:25, -0.15)
[3670.62, -0.77)
H (18))
50
[18)
Avg. Error
60
H(BB)
H [38)
H ((B))
H[SB)
40
TF (F(00)
(BB)
30
H(8B)
[[
40
50
1012
10
1012
10
DocVOA
InfoVQA
[675.70,
(10961.40, 0.39)
(70652.46, 0.30)
(2175.60, 00.3)
50
40
40
(100)
Avg. Error
00
30
(38)
30
(88)
11700)
[38
20-
40
20
50
1012
10
10
1012
DCI--bench
TextvQA
Realwordd
OKYQA (4158.34 -0.77)
(643.13.-0.177
(9056.43, -0.21)
(147329,.0.ll -O.13)
10
40
11311
H119]
H [35]
Avg. Error
40
H ((88)
30
40-
H (88)
30
40
100
1012
20
10
10
10
VQAv2
VizWiz
Avg. All
(1849.99, -..66
(480.40 40, C.09)
[2771.33, 0.166
28
42
[36)
Avg. Emot
26
HTB)
40
40
22
24
(30)
38
((B))
H (3B)
Power
(8B)
F1
20
34
30-
18
32
10
1012
10ka
GFLOPS
GFLOPS
CFLOPS
Figure 11: Synthetic Scaling Plots. Relationship between Average Error and training compute (in floating- point operations) for various 1B, 3B, 8B PLM with L14 vision encoder. Each plot reports the individual
error in VidcoMME [20], STAR [21], EgoSchema [22]. How2QA [8], MVBench [23], Perception Test [24], ChartQA [25]. Doc [26], InfoVQA [27]. OCRBench [28], RealworldQA [29], OKVQA [30], VQAv2 [31],
VizWiz [33], and TexIVQA [34]. Finally, report Avg. All, which average over all the metrics.
In Fig. 11, we show the breakdown of the scaling trend shown in $3.2. "H" stands for human only (i.e., no synthetic) baseline. From the breakdown, the most notable point is the the scalability in OCR, Chart, Document QA tasks. In each benchmark, synthetic data makes more than 10 points of improvement on every model scale, compared to "no synthetic baselines. Moreover, there is no sign of saturation; the performance will most likely improve with more synthetic data. We hypothesize that OCR, Chart, Document QA tasks reduce to "translation" task - set of pixels has one-to-one mapping to text space. Remaining tasks exhibit clean power-law relationship between metric error and FLOPs. The last plot shows scaling trend average over all benchmarks, which shows close power-law relationship.
30

---

C VLM Benchmark Details
In this section, we provide details about all the image and video benchmarks considered in $5 including composition and evaluation metrics for image benchmarks ($C.1), video benchmarks ($C.2) and our PLM-VideoBench ($C.3. We also describe evaluation protocol for all these benchmarks including inference parameters and prompts ($C.4). Pointers evaluation code linked where available.
C.1 Image Benchmarks
Image captioning We evaluate single image captioning and grounded image captioning bench- marks like COCO [101], nocaps [102] and Flickr [103]. We report CIDEr as the evaluation metric.
Perception and reasoning We evaluate on broad, general purpose VQA benchmarks like MMMU [68], VQAv2 [31], MMBench [94], OK- [69], VizWiz [33] as well as hard per- ception benchmarks like BLINK [71], CV-Bench [72], RealWorldQA [29], and VSR [73]. For all MCQ benchmarks, we report accuracy selecting the correct option.
Charts, diagrams and documents We evaluate on benchmarks for reasoning over various types of charts, graphs, diagrams, infographics etc. Specifically, Doc [26], ChartQA [65], TextVQA [66], InfographicsVQA [27], A12D [67], OCRBench [28], and SEED [70]. We report accuracy of selecting the correct option.
Image Hallucination Finally, we evaluate on benchmarks that evaluate robustness of models to hallucinated details in questions such as HallusionBench [111] and POPE [32]. For HallusionBench report the aAce metric (code) which accounts for correctness and consistency using an LLM judge.
C.2 Video Benchmarks
Video captioning We evaluate on short-video captioning benchmarks, namely YouCook2 [47] and VATEX [59] as well as recent detailed video captioning benchmarks - DREAM-IK [74] and AuroraCap-Via VDC [121]. For YouCook2 and VATEX, we report CIDEr score [208]. For DREAM- Ik we report AutoDQ Fl Fl-sc (code) and for AuroraCap-VDC we report the VDC accuracy (code) following the author's proposed metric.
Short video QA We evaluate on multiple-choice (MCQ) benchmarks such as How2QA [8], NExt- QA [50], PerceptionTest [24], STAR [21]. TGIF-QA [112], TVQA [113], Video-MME [20] and TVBench [117]. We report accuracy of selecting the correct option. We also evaluate on open- ended question answering benchmarks (wlo options) such as AetivityNet-QUA [51] (code), MMBench- Video [1]6] (code) and VCGBench- Diverse [84]. We report LLM- LLM-judge scores/accuracies for these benchmarks. For VVGGennch-Divert we report the average LLM-judge scores (code).
Long video QA We evaluate on popular long-video benchmarks such as EgoSchema [22]. LVBench [41], Long VongVideo [123] and MLVU [125]. We report accuracy of selecting the correct option.
Fine-grained video QA We evaluate on benchmarks for fine- ahe-ggrined spatial, temporal and detail rea- soning in videos such as TemporalBench [39], TOMATO [36], MotionBench [38], TempCompass [75] and CG- CG-Bench [35]. We report accuracy of selecting the correct option. For TemporalBench, we report the multi-binary accuracy (MBAcc) (code) proposed by the authors to reduce bias in evaluation.
We evaluate on benchmarks that evaluate robustness models hallucinated details in questions such as VideoHallucer [76] and EventHallusion [37]. We report accuracy of selecting the correct option.
Hallucination
C.3 PLM-VideoBee
We evaluate on our suite of benchmarks for fine-grained and spatio-temporal reasoning in videos. These include:
31

---

Fine-grained QA (FGQA) We report multi- binltiy accuracy (MBAcc) following prior work [39]. In short, this entails presenting the model multiple independent, binary-choice questions about the same video (in our case, three questions) and requiring the model to gets all of them correct, to count towards accuracy. This sets a higher bar for models, and combats bias in multiple-choice question benchmarks that prior work identifies.
SmartGlasses-Q2 QA (SGQA) We report LLM-judge accuracy of the predicted answer compared the ground truth answer. We follow existing LLM judge prompts from ActivityNetQA (code). The prompt repeated below for completeness.
Video Region Captioning (PLM-RCap) We use an L.MM-juge to generate the similarity scores between predicted and ground truth captions. The prompt below.
Dense Video Region Captioning (PLM-RDCap)) We adapt the SODA metric [60] from dense video captioning literature for this task. To compute this metric, we use the same LLj-- from above to generate the pairwise similiarity scores between predicted and ground truth captions, which is then fed the standard metric computation routine.
Region Temporal Localization (PLM-RTLoc) We report standard temporal localization metrics, namely Mean Recall I, averaged over a range of IoU thresholds [0.3, 0.5, 0.7, 0.9].
C.4 Evaluation Protocols
Common evaluation protocol. For video benchmark evaluations, we sample 32 frames uniformly from the full video unless otherwise specified. For uniformity and consistency across benchmarks, implement all LLM-judge evaluations using LLama3.3 3--0BB.--OBBB.. 1-7OB-Instruct [13], following LLM judge prompts from popular evaluation frameworks [209, 210] where available. Outputs from models generated via greedy sampling (temperature 0).
SG-QA judge prompt
You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. Your task to compare the predicted answer the correct answer and determine if they match meaningfully. Here's how you can accomplish the task:
#HINSTRUCTIONS:
Focus on the meaningful match be twenn the predicted answer and the correct answer. Consider synonyms or paraphrases as valid matches.
Evaluate the correctness of the prediction compared to the answer
Please evaluate the following video-based question-answer pair:
Question: [question]
Correct Answer: [target]
Predicted Answer: [candidate]
Provide your evaluation only yes/no and score where the score an integer value between and 5, with 5 indicating the highest meaningful match. Please generate the response in the form of a Python dictionary string with keys 'pred' and 'score', where value of "pred' is string of 'yes' 'no' and value of 'score' is in INTEGER, not STRING. NOT PROVIDE dictionary string. For example, your response should look like this: ("pred": "yes", "score": 4.8).
OTHER OUTPUT TEXT EXPLANATION. Only provide the Pythan
PLM-RCap judge prompt
Your task is compare given pair of captions provide a single score indicating how correct the pred is compared to GT, on a scale from to 10. Focus on meaning and context, not exact word matches. Penalize missing and incorrect information, with lower scores for more significant errors. High scores require accurate conveyance of all key information. Respond with only the score, starting your response with the number including no additional text. Output format: [score].
PidM-Vide inference prompts. Table 11 contains example inference prompt examples for each VideoBeeee task. Note that some variation exists between instances in the benchmark. For
32

---

example, for RCap prompt may be "What is happening to the subject in the region highlighted by the red rectangle ... instead of "Give detailed description the events occurring in the region marked by the red rectangle ...", however they convey the same underlying instruction and information.
Proprietary models like GPT-4o and Gemini require more careful prompting to ensure that the output formatting is respected. For example, we append instructions to prevent model hallucinations (e.g., "You must use these frames to answer the question; do not rely on any external knowledge or commonsense."), to prevent refusals to answer (e.g. "Even if the information in these separate frames not enough answer the question, please try your best guess an answer which you think would be the most possible one based on the question. Do not generate answer such as nor possible to determine") and in-context examples to help guide the model towards the correct output format. Model- and benchmark-speae inference prompts will be released along with our code for full reproducibility.
Task
Prompt
FGQA
Question: [question] \n Options: \n (A) [optionl] In (B) [option2] \n Only give the best option.
SGQA
The following question is asked by the camera wearer at the end of the video. Provide a detailed answer even if unsure. Try to answer in around 20-30 words. Now answer the following question based on the video content: [question]
RDCap
Create dense caption of the subject's s actions within the red rectangles, including action frames ids and brief descriptions. For each item use the format [start, end]: [description] separated by newline, where start and end are frame numbers between 0 and 31 in this 32 frame video.
RCap
Give detailed description of the events occurring in the region marked by the red rectangle within frames ([start frame], [end frame]) in this 32 frame video
RTLoc
Given the region marked by the red rectangle in the video, please provide the start and end frame of when '[event]' happens. Use the format (start, end), where start and end are frame numbers between 0 and 31 in this 32 frame video.
Table 11:
Bedc task prompts. Items in square brackets are placeholders filled for each benchmark instance.
D Baseline Implementation Details
We provide specifc implementation details for models in $5.1 of the main paper.
Proprietary baselines We evaluate the GPT and Gemini family of models. For GPT-4o, we use the 20T-40-2024-1--1 checkpoint. We feed 32 uniformly sampled frames regardless of vidco length, loaded at high image quality setting. For Gemini, we evaluate Gemini-1.5-Pa and (emmnni-2.0-Faasse For VQA tasks, we input the video (without audio) which is processed internally at fps. For spatio-tempordl tasks (RCap, RDCap, and RTLoc) we use the same inputs as for open-source models and GPT-4o. We evaluate these models using API call.
Open-source models We evaluate Intern Qwen, Molmo and Llava-OV models. We follow official implementation and preprocessing pipelines for each. Specifically, we evaluate VL2 and Intern VL2.5 (code); Qwen VL2 and Qwen VL2.5 (cade;; Molmo-O-0924 (code) and Llava-OV (code). For Qwen we sample frames at fps from videos. For InternVL.2 we use tiles per image as this more closely matches the reported results.
Human performance baseline. Table 5, we report human performance on PL.M VideoBBench. For each task, we present annotators with the test sets and collect answers for each instance given the standard task prompt. Given the difficulty of RDCap, we reuse our data annotation pipeline in &G to collect new dense captions independently, rather than providing the standard task instruction.
33

---

E
Additional Results
E.1 Additional PLM-VideoBench Results
We present benchmarking results across all model scales (IB, 3B, 8B) in Table 12, to supplement the 8B model results in the main paper (Table 5). Our approach consistently outperforms baselines across all scales, including proprietary models whose model scale is unknown.
S00A
8DCap
3012AY
Medel
Homan perf
90.9
67.9
66.6
53.9
67.8
70.9
Proprictary
GPT-4a [61]
20.9
35.7
Gemini 1.5 Pro 16a]
14,4
33.1
27.6
Gemini 20 Flsh [63]
58.7
44.8
13.2
30.9
27.6
42.5
1B scale
QwenZVL-2B [16]
0.9
IntemVL2-18 [10]
35.8
28.9
0.5
17.2
2.7
23.8
IntemVL22.51B [10)
PLM 1B
57.6
40.9
50.3
40.9
57.7
49.4
3B scale
VL-38
45.1
0.3
17.2
IntemVL2-48 (10g
43.2
41.7
0.5
19.9
9.5
30.3
IntemVL22.54 4B [10]
50.0
4,9
25.9
15.4
35.3
PLM-38
38.8
53.1
45.0
58.2
scale
LILWA-OV-7B [64]
40.2
41.5
4.7
24.4
13.9
32.0
Qwen2VL-7B [16]
Qwen2.5VL-7D [15]
49.8
43.0
2,5
21.5
10.7
34.8
IntemV1.2.-88 [10]
53.7
48.3
5.7
26.1
8.8
38.5
52.8
46.6
Table 12: PLM-VideoBench results across all model scales to supplement results in Table 5.
E.2 Comparison with LLaMA-3V
DovVQA (60
pood ooc (37)
ext [08]
(o00 oee [31]
ChanQA
(63)
TouVQQ
oee (66)
oxc (67]
MMMM
VQ4+2
ADD
Model
(11))
79.7
50.7
3.2V (98B) [13]
76.6
90.1
85.5
82.3
67.2
92.3
60.3
78.1
PLM-3B
74.4
93.8
84.3
84.3
74.6
90.9
41.2
84.3
PLM-3B
85.6
Table 13: PL.M LLaMA-3V Image Benchmarks: Note that we LLaMA-3V-90B [13] for generating
image captions our synthetic data engine.
E.3
Image Captioning
Table 14 shows PLM versus proprietary models and open-access baselines of comparable scale on Image Captioning benchmarks.
E.4 Image Grounding
Table 15 show PLM versus SoTA model on RefCOCO/+/g. PL.M performs competitively compared the baselines across all model scales, and outperforms specialist models.
E.5 Long Video Understanding
Table 16 show PLM versus open-access baselines and proprietary models of comparable scale, and report results over 3 long video QA benchmarks.
34

---

Gurpulay
CIDE [101]
CID 1122]
CIDSs (105]
CDCD
Nocap
Model
Proprictary
GPT-4o
Gemini 1.5 Pro [63]
70.6
71.1
68.2
Gemini Flash
66.6
IB scale
Qwen2VL-28 [16]
107.1
101.2
86.0
1ntemVL2..-18 (10)
122.6
110.5
86.1
PLM6-18
100.5
Owen22 VL-38 1151
101.7
105.5
77.5
IntemVL2.5-48 (10)
PLM-33
144.9
126.5
96.0
8B scale
11.NA-OV-7R
55.7
Qwes2.5VVL-8 [15]
36.8
32.7
34.9
IntemaVL2.588 (10)
96.5
PLMM-88
146.7
129.9
105.6
Table 14: Image Captioning benchmarks. PLM versus proprietary models and open-access baselines of
comparable scale Image Captioning benchmarks.
ReiDO0o
ReCDDOD
ROCDDOO
ReIDOO
ROCDDD
ReC0000
ReC0000
Specialists
OnvndingDNNO [2[1]
90.6
93.2
58.2
88.2
89.0
75.9
86.1
87.0
86.6
UNINEXT-I
89.4
ONE-PEACE [213]
90.6
93.2
53.2
53.2
89.0
75.9
86.1
$7.0
86.6
1B scale
91.5
$4.8
83.2
76.5
86.0
86.4
3B scale
Qwen2.5 VL-3B [15]
89.1
91.7
$4.0
$2.4
$8.0
74.1
85.2
85.7
85.0
PLM-3B
93.3
94.9
89.5
89.8
93.6
84.2
98.5
90.9
90.9
8B scale
Cube-TII [214]
S0
86
Qwen2VL-78 [36]
91.7
93.6
87.3
85.8
90.5
79.5
87.3
87.8
87.9
Qwen2 [15]
InterVVL2-88 8B [10]
$7.1
91.1
50.7
79.8
87.9
71.4
82.7
82.7
R2.9
TrternV1.2.5608 [10]
PLM-SB
90.6
91.8
83.9
87.3
91.3
81.1
88.8
89.2
88.2
Table 15: Image Grounding results on RefCOCO/+/g. PLM performs competitively compared to
the baselines across all model scales, and outperforms specialist models for the image grounding task.
E.6
Additional Ablations
Table 17 shows the performance PLM under full stage-3 setup, and Table 18 shows the effect of removing in-distribution data in PLM- VideoBench.
35

---

Long Video
VongVido
(dev)) Morg (125)
(ni) (na)/o [123]
I-PRach
[41)
Model
Proprictary
GPT-4o [6]]
66.7
Gemini 1.5 Pro [63]
33.1
64.0"
69.9
Gemini Plash [63]
61.6
1B scale
Qwen7N1-228 [16]
42.0
47.9
IntemVL2-1B [10]
31.4
43.3
52.0
Im..VL225118 [10]
57.3"
PLM-IB
40.0
$2.3
58.9
3B scale
54.2
IntenV1 248 (10)
34.0
53.0"
59.9"
ImtmVL225-8 [10]
40.1
56.3
68.3
PLA--R
40.4
57.9
63.0
8B scale
LLaVA OV-7B [64]
38.8
55.7
61.6
Qwen2VL--B [16]
46.0
53.8
69.8*
2wen2.5VVL-7B [15]
45.3
$6.0"
70.2"
IntemVL2-88 [10]
64.0"
IntemVL2.5 SB [10]
43.2"
60.0
68.9
56.9
66.4
Table 16: Results on long video understanding tasks. We compare PLM with open-access baselines and proprietary models of comparable scale, and report results over 3 long video QA benchmarks. Cells with * are reported numbers from literature. The remaining are reproduced using official code.
Bodd aod (26)
(ovt (27)
(vl)) oce (69
(mand amage)
Redl WarkOOA
Ba-YQA
ClunQA doc (65)
TesIVQA ooe (66)
Into
molomas)
aac (215)
OCRBendh
acc (28)
VQ-v2
are
OK VQA (10)
VizAWi
doc (23)
(mmogee)
(X0)
BLINK
acc (7
BesBe
aod (29)
acc (73)
(32)
AZD
Model
Stage243
91.8
82.6
62.8
78.5
PLM-30 3B
76.5
93.8
$4.3
$4.3
34.6
90.9
83.0
41.2
84.3
56.8
64.0
78.5
55.4
81.4
72.4
80.4
88.7
MVBendh
Penepp Teat
VidrMME
AcNooQA
EgoSehemm
(est) (22)
TampBench
TOMATO
MotionBendi (de)) ace (38)
Compoompes (M02)acc ace
CG Deach
Charatee STA
md00 (40)
Video overal oce 196) (96)
acc (37)
acd (25)
ake (50)
STAR
(211
ace (20)
acc (51)
Milier
acd (36)
Ave
S1age2+3
71.6
84.4
56.7
23.5
PLM 3B
62.5
74.7
83.4
79.3
$4.8
54.9
66.2
72.6
23.4
30.9
60.4
69.3
47.
57.7
55.5
76.5
Table 17: Performance comparison under full stage-3 setup. We run PLM-3B using combined 90M+ data blend with up to 36 image tiles and 32 video frames (i.e., stage-3 configuration) and compare against other open multimodal models. PLM-3B achieves the best overall performance localization benchmarks, highlighting its superior generalization and cross- understanding.
reasoning, captioning, and
REQA
5004
S0D4F
RCap
sc0rc
Model
Qwen2.5 VL. (3B)
43.7
45.1
0.3
139
24.0
InternVL.2 (4B)
50.0
49.2
154
29.1
M (38, no ham)
$4.0
41.5
11.1
27.6
25.0
318
Table 18: Effeet of removing human-annotan annotated in-distribution data in PI.M-VideBBench.. We retrain
PLM-3B after excluding all auman-annotated samples from in-distribution datasets, Despite this stricter setting. PLM outperforms Qwen2.5-VL (3B) and Intern VL2.5 (4B) on most metrics and achieves the best overall average, indicating stronger generalization without in-domain supervision.
36

---

PLM-FGQA: Fine-grained QA
We present PLM-FGQA Fine-grained QA (FGQA), a video dataset focused on "how" actions are performed, capturing nuanced fine-grained details through specially designed questions and carefully annotated answers. Due to the scarcity of fine- tne-grained video Q&A data, see Table 19, we built data engine to enable the collection of our 2.4M Q&A dataset, PLM-FGQA. As illustrated by the breakdown of question types' Fig. 4 (top-right), PLM-FGQU contains a large number of annotations about fine-grained details that have been largely missing in existing training video QA datasets [24, 49-56].
Dataset
Year
#Q&As
Dataset
Year
#Q&As
MovieQA
2016
6462
STAR
2021
60000
MSRVTT-QA
2017
243690
CLEVRER
2023
82620
TGIF-QA
2017
165165
EgoQA
2024
19000
MSVD-QA
2017
51000
Perception'est
2024
TVQA
2018
152545
VideoInstruct
2024
25803
ActivityNetAA
2019
58000
MoVQA
2024
21953
How2QA
2020
44007
CinePile
2024
303828
NexT-QA
2021
52044
Sports-QA
2025
94000
PLM-FGQA
2025
2379067
Table 19: Comparison of our PLM-FGQA dataset with existing video-QA datasets.
F1 Annotation process: Data Engine
Our data engine is built upon the following modules: (1) Temporal Segment Generation, (2) Question Generation, (3) Answer Generation, (4) Human Annotation (answer verification/manual answer annotation), (5) Quality Control, as illustrated in Figure 12. Next, describe each module in detail, and finally also provide additional details about the extra steps we took for forming the FG-QA out of these annotations.
component of
Temporal Segment Generation
Question
Answer Generation (PLM)
Human Verification & Correction
Video
Generation (LLM)
FCQA
Figure 12: Data engine used to collect the PLM -GM- dataset.
F.1.1
Temporal Segment Generation
We source the video data that serves basis for annotations from publicly available datasets. Based on the video sources and the type of existing annotations, we split the videos into three distinct categories.
Videos with existing ground-truth segment annotations: We directly adopt segments with their human-annotated action annotations from the following datasets: Ego4d Goal-Step[216], Ego4D Moments[43), EgoExo4D [44], HTT-Step[277, 218), COIN [45], CrossTask [46], and YouCook2 [47]. All those sources provide video segment boundaries accompanied by some form of textual action descriptions, and are therefore readily usable with the rest of the pipeline.
Unedited videos of physical actívities: For physical activities videos (e.g. basketball, dancing, soccer), actions are usually atomic and short (e.g. dribble, dance move, kick) and therfore reuqire precise temporal localization. To source videos for these scenarios we used data from EgoExo4D [44] that contains temporally well-aligned and precise narrations; we obtained segments of 2-3 seconds centered around narration timings, and used the anchor narrations directly as the action description.
Raw, untrimmed videos in-the-wild without temporal segment annotations. We source very large part of our data from untrimmed instructional videos the large-scale HT000MM dataset [42] which we first need segment before use. The goal is to obtain video clips that contain meaningful,
obtained with LLM-based tagging.
37

---

Speed Force
Action
Spatial Relation
4.0%
15.5%
2,
YouCook2
5.0%
Object State
HT100M
8.1%
17.8%
37
Other
11.35
Obj. Location
7.4%
Pose 9.8%
EgoExo4D
Obj. Recognition
8.0%
18.2%
8.1%
Counting
Obj Attributes
Ego4D 20.7%
8.9%
Figure 13: Distribution of question types (left) and video sources (right) the FGQA component of PLM-VideoBench.
salient actions, and also caption the resulting segments with concise but accurate action descriptions. We describe the automatic segmentation and captioning module in the following.
The automatic segmentation and captioning pipeline involves the following three stages:
Temporal segment proposal. Given untrimmed long videos, the first step is to identify semantically coherent segments within them. Inspired by prior work on unsupervised action proposal and seg- mentation, we leverage visual feature clustering to generate temporal segment proposals, and use shot-boundary detection results further refine the segment boundaries. We extract clip-level visual features(219) using sliding window with temporal stride similarity between neighborhood features and detect the class-agnostic action boundaries using boundary detection kernel (similar to those used in iiteratue[220, 221J). Finally, since the detected segments usually over-segmented, we perform bottom-up agglomerate clustering approach to group adjacent segments into clusters, using segment duration prior 10 seconds. We also leverage shot boundary detection[222] to obtain precise moments of scene changes: we refine the boundaries of the segment proposals by aligning them to the detected shot boundaries when they re sufficiently close (S second).
second. We then compute the pairwise
Segment filtering and ranking. How-to videos often include a lot of content that irrelevant to the demonstration of the activity hand, such the instructor explaining what they about do showcasing tools and ingredients. It is therefore important to detect and filter segments with such uninformative content. To that end rank candidate segments according relevance using series of heuristies and learned models, described bellow.
Talking head detection. A common mode in instructional videos is instructors talking into the camera, describing objects or explaining actions they re about to take. To detect and remove such segments, we employ Active Speaker Detection (ASD) pipeline[223]] which every video and combine resulting talking head tracks, to produce an ASD score for every segment.
densely
b. Hand-objer interaction (HOI) detection. The presence of hand-object interaction (HOI) can be good indicator of visually groundable actions. We leverage the temporal selection strategy[224 to filter out the segment proposals that contain hand-object interaction. We first employ off-the-shelf robust HOI [etector[225] to densely extract HOI regions within a proposed segment. The HOI score is then calculated by measuring the likelihood of hand-object interaction in the segment and the averaged probability of all the detected hands.
ASR groundability. HT 100M videos contain timestamped ASR captions, which are speech transcriptions of the audio instructions. It is desirable rank candidate segments based on how likely their ASR content is to their video content. The hypothesis here is that segments containing ASR transcriptions that align well to the video content, are more likely to be issualinffrrmaiion rich. Moreover since the action labeling pipeline (described next) relies on ASR metadata for producing descriptions, higher ASR groundability scores make it likelier to produce good quality segment descriptions. For every candidate segment, we compute an ASR -So-ground score by computing video-text alignment scores[219] for each ASR caption within the segment and then averaging the ones that above threshold (we use 0.5).
38

---

d. Relevance classification. The above heuristics work well for the clear-cut cases they are tailored for, but practice we found that they struggle with more nuanced segments (e.g. instructor fiddling with an object and describing it rather than using it). To improve the detection of those cases, we manually labelled a small amount segments that passed through the other filters and trained a binary classifier to classify them as "relevant" or "irrelevant"; to that end we trained simple 2-layer MLP classifier on top of temporally pooled video representations with logistic loss for binary classification. We deployed the trained model to provide a relevance score for all the candidate segments.
We combined the scores resulting from all the modules described above and determined cutoff thresholds, based on small manually annotated validation set. In production, we keep all the segments that have relevance scores above those thresholds.
Segment captioning We follow a two-step process to obtain action labels for each unlabeled segment: In the first step, collection of off-the-shelf perception models are used extract individual image- level captions, video-level captions, and object detections from the segment. The output of all perception models is then fed text into an LLM to generate long, fine-grained captions. At the second step, the detailed captions are fused with the ASR content of the segment, to obtain a consice action description. Specifically, we query an LLM (Llama 3.3 70B [13]) with the following prompt:
Segment action labels prompt
Detailed description: [fine grained caption] ASR transcription: [asr caption]. Given the detailed description above, identify the specific action performed as part of the activity [task name]. Your response must not be the same as the activity [task name] and needs to be specific substep within the activity [task name]. Please also supply a rationale for your answer.
The extracted labeled video segments obtained through the above process serve the foundation for the subsequent Q&A generation.
F.1.2 Automatic Question Generation
We automatically generate questions about the fine-grained details of the way activities are executed the video. Our questions generated with variety of prompts and models which lead increased question diversity and specificity. In Table 20 we present the question types and sample questions per question type. Here, we summarize how these questions are generated automatically with an ensemble with models and prompts:
LLM-based aetion-conditioned question generation Given segment, its action name (e.g., cur potatoes), a task name (e.g.. How to make sweet potato gratin) and optionally other metadata about the segment (for example, recognized objects [ }). we generate questions that can elicit descriptions of fine-grained details by raters with an LLM. We use tailored prompts for generating questions that cover how the activity is executed (tools, object locations, object states, direction of movements, hand pose), and the spatial arrangement of objects.
Activity FG question generation prompt
I learning how to [action name] while [task name]. . Ask [N] most relevant questions
that reveal the details of the way the step is executed in my environment, e.g., (a) part location, (b) types tools/ingredients used, (c) direction of movements, (d) how are objects held, (e) object states at the beginning of the step, (f) object state
at the end of the step. The questions must be answerable by visually observing the activity, wi thout reading instructions
trying out. Please indicate the type of
question from (a) to (f) for each question asked the beginning of question.
Spatial FG question generation prompt
Imagine I have common sense understanding of the 3D real world. I trying to [task name]
the step where am [action name]. There's [object list] when I'm [action name]. Ask me [N] questions about the 3D position objects, relative location between objects, distance between objects, spatial relationship using prepositions like above, below, next to, etc. that I might want to know. The questions must be answerable only visually observing performing activity, without reading instructions or trying out.
39

---

We explicitly encourage the LLM to provide questions that can be answered solely based on the video frames, in contrast questions that are focused on external knowledge or non groundable concepts or judging the execution of the step (e.g., avoid questions like is the pan hot enough to add the oil?, what tool rypically used loosen the axle nut). The rationale for this is to collect as many Q& prior, but they rather
pairs that a model cannot answer just based on external knowledge/languee require vision perception to be answered. Note that these questions are generated without visual input, hence they are not instance-speciffc and might not be answerable given the video segment.
VLM-based instance-specific question generation After collecting a first set of Q&As using the LLM-generated questions, we bootstrap a VLM Question Generator model, which takes as input the video segment, question types and optionally the task name, and generates set of instance- speciiffc visual questions. The VLM Question Generator model is obtained by supervised fine-tuning of PLM with a question generation instruction-tumi tuning dataset which consists of triplets (video, prompt, response), where the prompt includes the instruction to generate questions based on question types and the response includes example questions be generated for the given video. Due to the lack of such a dataset with fine-grained question, synthetically generated it by utilizing the Q&A pairs obtained based on the --M-generated questions. Specifically, for each video segment, we use an LLM to (I) decompose existing Q&A pairs into multiple Q&A pairs, with each new question focusing on one detail the original answer; (2) tag question types for the generated questions based on expanded list of question types; and (3) generate (prompt, response) pair for the segment. This resulted in 600k training instances.
VLM Question Generator training sample
Generate di different questions that reveal the fine-grained details of the way the activity executed. locations, hand pose, object/repetition counts, generating at least one question per type. Write each question in a separate line, e.g., Q1. first question. Q2. second question.
particular, focus on these question types: fine-grained object
QN. N-th question.
Response:
Q1. Where are the tomatoes positioned prior to being cut?
Q2. How is the person grasping the tomato with their left hand?
Q3. many tomatoes did the person use the segment?
LLM-based follow-up question generation This final set of questions aims to increase coverage of video details and generate highly fine-grained questions by leveraging the already collected Q&A pairs for each segment and feed them to an LLM that generates "follow-up" questions that are more detailed and challenging than the initial questions.
Follow-up question generation prompt
I have the following information gathered about the video: [list of previous Q&A samples] Utilizing information and details from all the provided Q&A pairs (make sure to specialize questions based the already corrected answers, e.g., using referring expressions),
[N] most relevant and interesting, visual questions that we can ask annotators order reveal NEW, rich, additional fine-grained details about the video that we don't know yet, in particular about the following question types: ttools/ingredient'. 'object counts', 'repetition counts', 'direction of movement', 'hand pose', 'fine-grained object locations', 'spatial relations', 'initial state/end state', 'action happened before/after', 'clothes wearing', 'body pose', 'main action in the video', 'temporal extent of action', 'sizes'. The questions should be specifit and have a specific answer. Avoid generic questions that can be very tedious to answer, e.g., how many objects are there in the scene. Also, do not generate questions that start with "Is ... and then list options. Prefer open-ended questions, e.g., starting with "How". [... More examples formatting ...]
F.1.3 Automatic Answer Generation
The next step the data engine aims to produce correct and comprehensive answers to the generated questions. We obtain automatic answers to the generated questions using version of PLM that has been fine-tuned with extra privileged information of various forms as input. The privileged information includes textual annotations from the metadata available with the candidate training
40

---

Question Type
Sample Questions
Action Recognition
What is the process being performed on the sandpaper?
What is the action shown?
Action Sequence
What does the person do after brewing the tea?
What does the person do before marking the vinyl with pencil?
Counting Problems
What is the quantity of universal down cleaner being poured into the task area?
How many branches does the person cut in total?
How many times does the person spray Greased Lightning onto the ketchup spill?
Movement Direction
what direction is black welding tool pointing while the person is working on the metal joint?
How does the person chop the garlic with the knifc?
Object Attributes
What is the color of the scatpost shown in the video segment?
What is the shape of the tube at the end of the step?
What is the size of the knifc being used to chop the spring onions?
Object Location
Where does the person put the honey bottle away?
Where does the person position the clothes before ironing?
Ohject Recognition
What type of roller and paint are being used?
What does the person place top of the smooth half of the egg carton?
What was the person initially holding in their left hand?
Object State
How would you describe the sink at the beginning of the cleaning process?
What is the state of the nematode after mixing with water and sponge?
Other
At what point in the video is the person seen holding the wires?
Pose
How are the woman's 's legs positioned while she is sitting?
How bent is the left elbow during the activity?
Spatial Relations
How far is the bias tape maker from the right edge of the ironing board?
What is the spatial relationship between the bowls and the Brussels sprouts the kitchen countertop?
Speed/Force
How would you describe the consistency of pressure applied during sanding? How fast does the person initially push the stone?
Table 20: PLM-FGQA question types and sample questions
videos and feature embeddings extracted from off the-shelf models. Useful textual metadata include the video title, ASR captions written descriptions, video-level task name (infered by an LLM using the title and captions), and any existing QAs for that video. Off-the-shelf embeddings include frame-level features extracted denseley at fps; we an open-vocabulary object detection model, OWLv2 [226], for embedding object detection information and CLIP ViT-L14 embeddings [227] for scene classification information. We incorporate the textual annotations directly into language prompts using the following template:
Automatic answer generation prompt
A video is showing task [video level task name], specifically the part where [ASR caption]. Here what we already know about the video: [existing question-answer
pairs]. Answer this question in detail: [question]
The thf-the-shelf embeddings are incorporated into the PLM input via an additional Perceiver-10]228] tokenizer, which summarizes the embeddings at the segment level.
We fine-tune the answer generator on IM manually annotated QA pairs. After fine-tuning, we deploy the trained answer generator with privillged information access on the unlabelled questions produced in the previous step. to produce automatic answers.
F.1.4 Human Annotation
After obtaining segments and generating questions and automatic answers, we employ human annotators to obtain high-quality answers. Our answer annotations include the following:
41

---

Human-verifer answers: Raters are provided with the model-generatee answer and are asked to accept or reject the answer. They can reject questions for being irrelevant or unanswerable, and answers for being factually incorrect or lacking details. Accepted question-ansee annser pairs proceed without changes, while rejected ones are handled differently: question-related rejections (irrelevant or unanswerable) are discarded, whereas answer- related rejections (factually incorrect or lacking details) are marked for correction in the next phase. 17.8% of the total training samples are human-verified automatic answers.
Human annotated answers: Raters answer the questions from scratch by ensuring to cover all the relevant details within the temporal segment. They receive reference information, such as video-level task names and ASR captions, and may use online resources like WikiHow for additional context. Questions that cannot be answered based on the video segment (for example, due to some false premise) are rejected (with an explanation). These manually annotated answers make up 82.2% of the PLM-FGQA training split, and 100% of the evaluation set.
Data quality is crucial for model success. We followed several strategies to monitor and enhance annotation quality: annotation Certification we reviewed small sample of annotations from each rater before they could work in production queues, ensuring that annotators met high-quality standards before advancing to production; golden Examples annotators were provided with high-quality annotation examples, highlighting common error patterns and offering acceptable answers. targeted and Dual QA we conducted daily audits, including vendor auditing and our own sampled quality control. In total, 13% of the training set audited, and 100% of the samples in PLM-VideoBench underwent quality control.
Quality Control.
F2 FGQA PLM-VideeBench Construction
Train
Test
Sources stats
Total Vidcos
767k
3.6k
Unique Source Videos
251k
1.9
Average Duration (sec.)
9.8
12.3
Annotations stats
Number of QA Pairs
2.4M
4.2k
Number Question Types
12
12
Question Length (avg/max)
12/114
12.3/56
Answer Length (avg/max)
13.3/911
14.1/62
Annotation Type
Human Yes
Human Yes
Table 21: Statistics of the PLM-FGQA training and test data. module of PLM-VideoBench.
The test split refers to the FGQA
The FG-QA component of PLM-VideoBench is formed from a held-out portion of PLM-FGQA. We refine this set and transform into a challenging MCQ-based benchmark by (1) generating MCQs, (2) filtering out samples that can be answered by text-only (blind) LLMs, (3) performing human verification of negatives, and (4) balancing the distribution of question types and domains. The statistics of the dataset are summarized in Table 21. In more detail the steps we followed are:
MCQ Generation: To transform QAs into challenging MCQs for evaluation, instead of generating random incorrect answers, we prompt L.LMs to produce hard negatives that are semantically close to the correct answer. We use the following prompt which was designed generate distractors that differ from the correct answer by only single detail. In effect this enables evaluation to assess fine-grained reasoning about object attributes and tool distinctions.
Filtering Text-Only Answers: To ensure that hideo-based reasoning is required, we test whether text-only LLM can answer the question correctly without seeing the video. If question can be answered correctly from text alone, we remove or modify it to emphasize visual and temporal grounding.
Human Verification of Negatives: Automatically generated negatives may sometimes factually true despite being labeled incorrect. To address this, we perform human verification, where annotators
42

---

review distractors to confirm that they are both plausible yet definitively incorrect given the video context.MCQs with ambiguous distractors are removed.
Balancing Question Types:
Finally, after the above postprocessing and filtering is done, we rebalance the test set, make that the question type and domain distributions approximately uniform, by undersampling over-represented qyestion types and domains.
Note the evaluation metric. We report the multi-binary accuracy (MBAcc) [39] evaluate on the FG-QA task. This accuracy calculated by comparing the correct answer to each distractor individually. Specifically, for each question, correct answer is compared with one distractor at a time. A prediction is considered correct only the correct answer is consistently selected across all binary comparisons. We preferred this metric to vanilla MCQ accuracy as greatly reduces the predictability of automatically-generate MCQs.
generate series of binary questions, where the
MCQ generation prompt
Here is question and answer pair about video:
Q: [question]
[answer]
You need to transform this into high-quality multiple-choice question. do this, first rephrase the given correct answer and then provide n distractor answers. The n incorrect answers should be reasonable and valid responses to the question, but should have different meaning than the correct answer. You generate an incorrect answer from the correct one by changing single detail, e.g. object or verb/action that relevant to what's being asked. Make the incorrect answers realistic, plausible and similar enough to the correct answer so that it is very difficult for someone to distinguish between them with prior knowledge alone. Finding the correct answer should also require visual information about the scene. The distractor answers should answer question, but should be incorrect but a non-obvious way. When changing a single detail to create the distractors, make sure that this detail is the main point of the question. For example, if the question is about the color of an object, then the
distractor should change the color of the object and not the kind of object. Here some examples of good distractors (desired) and bad distractors (to be avoided):
Q: What is the person wearing on their hands while applying varnish?
A: The person is wearing white gloves their hands while applying varnish with brush.
Good distractors:
The person wearing black gloves on their hands while applying varnish with a brush.
Bad distractors:
The person is wearing black gloves their hands while applying paint with roller.
More examples formatting ...
43

---

G PLM-STC Details
We present PLM evaluating VLMMs for spatial-temporal reasoning. We collected pairs of mask tublets for objects in videos, along with their corresponding detailed temporal descriptions. The annotations are collected on top of the SA-V [57] videos, which diverse and high-quality. We excluded the test set videos from SA-V V, to avoid any data cross contamination. Table 23 provides statistics about the dataset, such number of total samples,
Captions (PLM-STC), novel dataset aimed at training and
teainiin/vall//ese splits, object types, and time-segment duration. PLM-STC, not only novel, but also larger and higher quality compared to existing datasets, see show an example of our temporal captions, describing little girl (highlighted in blue): (frame 0-81): A little girl moves back as beluga whale approaches her Jace. (frame 82-85): Out frame. (frame 86-98): She tries to feed the whale.
Table 22. In Fig. 5 (right),
Dataset
Spatial Type
Year
#Videos
Regions
Temp, Seg.
Captions?
DAVISI6 RVOS [229]
Segmentation
2018
50
50
No
DAVIS17-RVOS [230]
Segmentation
2018
90
205
Nn
YouCook2-BB [47]
BBex
2018
647
4.3K
No
A2D Sentence [231]
Segmentation
2018
3.7K
4.8K
No
J-HMDB Seatence [232]
Segmentation
2018
928
928
No
ActivityNet Entities [233]
BBox
2019
14.3K
1.5M
52K
No
VidSTG [9]
BBox
2020
6.9K
44.8K
No
Befter-Yottbee- [234]
Segmentution
2020
3.9K
7.5K
HC-STVG [235]
BBox
2021
16K
16K
No
VLN [56]
Mouse Trace
2023
50K
4K.1K
43.1K
Yes
MeVis [236]
Segmentation
2023
2K
8.8K
No
PLM-STC
Segmentation
2025
45.7K
122.3K
194.2K
Yes
Table 22: Spatiio-Tempora--Caatiion datasets comparison.
We describe the overall annotation process Appendix G.I, and how build the three sub-tasks in Appendix G.2.
G.1 Annotation Process
Object Selection Tracking
Temporal Seg. & Captioning
Verification & Quality Control
Spatio Temporal Captions
Video
Figure 14: PLM-STC Annotation pipeline.
The annotation process is summarized in Figure 14. The annotation process involves three stages: Object Selection and Tracking, Temporal Segmentation and Captioning and Verification and Quality Control.
G.1.1 Object Selection and Tracking
Annotators select interesting objects with significant motion changes in the video and use SAM 2 [57] to generate initial mask tublets, which they then refine to ensure high-quality spatial-temporal segmentation. We instructed the annotators by defining interesting regions in video footage those with the presence of significant, dynamic actions performed by subjects, which can be human, animal, or object. These regions involve multiple major actions that evolve over time, rather than static or insignificant actions. We provided annotators with examples of interesting regions, such as one featuring person making sandwich, dog chasing a cat, or kite getting stuck a tree. The goal for the annotator is to identify regions with high delta, where the subject performs a sequence of significant activities that change over time, such as a person entering a room, sitting down, and then drinking from glass. By focusing on these dynamic and evolving actions, annotators can effectively select regions worthy of captioning. Finally, annotators are provided with several examples of good and bad annotations.
44

---

G.1.2 Temporal Segmentation and Captioning
Based the selected mask tublets, another set of annotators provides time segments for each action and fills in the caption within each time segment. The annotators are instructed to focus on capturing major actions, avoiding minor details unnecessary movements. When writing captions for each segment, they must ensure clarity in describing the subject's movements and directionality. Additionally, the annotators are advised to avoid making assumptions about the subject's actions adding details not clearly visible, sticking only to what is directly observable in the frame. As in the previous task, the annotators provided with several examples of good and bad annotations guide their work.
G.1.3 Verification and Quality Control
A final set of annotators manually verifies the tublets and time-segment captions to ensure accuracy and consistency. For mask refinement, we re-run the same pipeline as $G.1.1, while not letting the annotators choose the interesting object, but only refine the quality of the mask. For captioning refinement, the annotators are tasked with three objectives: I) Redundancy: eliminate any repeating redundant information to ensure the caption is concise; 2) Accuracy: verify that every word in the caption accurately describes a fact present the video, correcting or removing any incorrect information; and 3) Actions: add missing major action information the caption while preserving existing atomic actions, ensuring the caption effectively conveys the key events in the video.
G.2 PLM-STC Benchmark
All
Train
Val
Test
Dataset stats
Number of Vidcos
45.2K
42.0K
804
2.3K
Spatio Temporal Caption
127.8K
Temporal Caption
198.7K
Tube'ss categories
Person
104.5K
99.6K
861
2.4K
Animal
16.8K
13.2K
550
1.7K
Object/things
6.4K
4.4K
436
1.2K
Temporal captions per Tube
caption per tube 2 caption per tube or more Caption per tube
78.9K
73.9K
842
2.4K
30.9K
27.8K
566
1.7K
16.38K
14.15K
421
1.2K
Tasks stats
Region Detailed Captioning (RDCap)
122.3K
117.2K
2.5K
2.6K
Region Captioning (RCup)
194.2K
179.5K
4.6K
10.1K
Region Temporal Localization (RTLoc)
192.0K
179.5K
4.6K
7.9K
Table 23: PLM-STC dataset statistics. Note the for RTLoc, we filter the test set to include only the captions that are unambiguously localized, i.e., they map to a single time window in the video. As a result, the test set size is reduced to 7,910 instances compared to RCap.
We utilize the collected data train and evaluate the PLM on three challenging tasks that essential for video perception. Firstly, we created balanced validation and test split by the combination of tube categories and number of caption per tube while making sure video overlaps with the training set. This is done to make sure we evaluate all the categories presents in the dataset equally. Then, we process the data for each task:
Dense Video Region Captioning (RDCap). This comprehensive task combines both "what" and "when" aspects. The model takes the video and the tubelets as input and outputs the full time-segment captions. We also assign an out of frame caption to temporal segments for which the subject does not appear in the video to ensure dense temporal coverage of events across the video duration.
Video Region Captioning (RCap). This task involves describing "what" activities are performed within a specific time frame by the objects in the tubelets. The model receives the video, the tubelets, and the temporal region as input and outputs the corresponding captions. We filter out events that refer to the subject when it is out-of of-mmfmm to avoid evaluating trivial captions.
45

---

Region Temporal Localization (RTLoc). This task requires the model to localize "when" specific events occur relation given tubelet. The imput includes the video, the tubelet, and the caption, while the output is the start and end frames indicating when the captioned event occurs. Like RCap, we filter out out-of-frame events, as well as ambiguous events that may be localized multiple time segments. For example, if the subject opens the door twice, the event text are guaranteed to be unique (c.g., referring to the first and second time they opened the door) or dropped entirely ambiguous (e.g., if the text only mentions the action).
These tasks are designed to both improve and evaluate the model's capabilities, with the same input- output format applied during both training and evaluation. Figure 6 illustrate an examples of the task, including the prompt used to train and evaluate the PLM.
46

---

H Smart Glasses Data
H.1 Data collection and annotation
We collected the source videos for PLM-SGQA using commercial smart glasses, which enable participants capture egocentric videos hands-free manner. Participants are presented with 14 categories of popular scenarios, such as shopping, cooking, and walking in a neighborhood, and are instructed to ask questions about their surroundings as if interacting with a multi-modal assistant that shares their visual perspective. Specifically, participants are asked to ask questions spontaneously, without delay, about the things they see and experience, and to focus on visual queries rather than dynamic information that may change regularly. After recording the videos, participants annotate the segments by marking the start and end points the video relevant to each question, as well as providing the ground-truth answer.
H.2
SGQA Benchmark
To create the SGQA component of FLLM-VdeeBent we first filtered the Q&As using an LLM to obtain a shortlist of questions that focus on human activity and also are perception-basee rather than based on general knowledge. This means that SGQA focus on questions that require good visual understanding of the scene to be accurately answered. This process yields evaluation set consisting of 655 As. For the resulting Q&As, we then trimmed the original videos to obtain clips within the temporal boundary that the human wearer/annotator specified. As the annotated segments end at the point where the smart-glass wearer asks the question, is important for all evaluations to specify that the question refers to the end of the video clip e.g. see the prompt we used for PLM and baselines evaluation 1I. We summarize the statistics of the SGQA test set in Figures 15 and 16.
Cooking
Shopping
Sources
Sports
3.5%
17.4%
Total Videos
663
4,2%
Average Duration (sec.)
29.4
Hobbies
Number of QA Pairs
Annotations
665
6.0%
Home DIY
Ind. 13.2%
Number Domains
14
6.1%
Question Length (avg/maxx
9.0/52
Fashion/ Style
Answer Lengih (avg/max)
21.6/
6.9%
Annotation Type
Human
Daily Activities 11.0% Dance Exercise 10.4%
Open-Domain Domain
Yes
7.8%
Sightseeing
Figure 15: Statistics the PLM-
9.4%
SGQA test data.
Figure 16: Domain distribution of video-clips in PLM-
SGQA.
47

---

I Synthetic Data Engine
Our data engine targets base capabilities of VLMs: image captioning, visual question answering. OCR, chart/diagram understanding, and video understanding. We developed different pipelines for images and videos, and includes different levels of metadata to generate captions and QAs.
Image Captions: We caption high-quality images using Llama 3.1V 90B. An example is shown in Figure 17. We use this pipeline caption SAIB [14], Object365 [135], and Openlmages [136].
OCR QAs: We leverage pre-extracted OCR and use as input for L.LM (i.e., Llama 3.3 70B) to generate set of five question-answer pairs. An example is shown in Figure 18. We use this pipeline to generate QAs for PDFAce [132], and UCSF [133].
Image Captioning plus QAs: In cases for which OCR does not provide enough information to create questions (e.g., scientific figures), we futher caption the image using Llama 3.1 90B. Then we pass the caption with auxiliary metadata (e.g., OCR) to LLM (i.e., Llama 3.3 70B) to generate question-answen pairs. An example is shown in Figure 19). We use this pipeline generate captions
and QAs for ArxivQA [104], DoA [26], Info [27] and Ai2d [67].
Video Captioning plus QAs:
An image captioner is run key-frames of the video, as well a video captioner on the overall video at 1 fps. The result captions are passed to LLM (i.e., Llama 3.3 70B, or Llama 405B) with additional metadata (e.g., video title etc.), so to generate detailed caption and a multiple-choices question answers pair. An example is shown in Figure 20). We use this pipeline generate captions and QAs for YT-IB [237], Ego4d [43] 4, DiDeMo [140], Charades [139], and Kinetics7 10 [54]
Detailed Caption
The image depicts a yellow building with red door and sign that reads "YOLCU SALONU" in black letters. The building has gable roof and small window on the left side. There are two trees in front of the building, each side of the entrance. The ground is wet, and there is the trees and the ground. In background, there is train station with a train on the tracks. The sky is gray and overcast. The image appears to be photograph taken on a cold winter day. The presence of snow and the gray sky suggest that it winter scene. The building and the train station in the background suggest that it public place, possibly a train station or a bus stop. The sign on the building reads "YOLCÚ SALONU," which is Turkish for "Passenger Lounge." This suggests that the building is waiting area for passengers. Overall, the image depicts quiet and peaceful winter scene, with focus the building and the surrounding environment.
Detailed
Image
Captioner
Captions
Figure 17: Detailed caption example, and the corresponding pipeline. The captioner (i.e., Llama 3V 90B) is prompted to generate the caption for the provided image.
'For dataset we used Llama3 405B, rather than Llama 70B 'DiDeMo, Charades, Kinetics710 used a simpler pipeline where only frame captions were used, and a smaller scale LLM (i.e., Llama 3.1 8B)
48

---

OCR
Ladies Division Three THIS ROUND TOTAL Dia- mond CLUB RINKS WIN FOR AGAINST MARGIN POINTS RINKS WINS FOR AGAINST MARGIN POINTS Earned The Albert 2 0.5 57 57 6.5 10.5 4.5 315 276 39 36 525 Coolum Beach 0.5 57 0.5 7.5 3.5 312 274 38 375 Mooloolaba 2 22
Ladies Division Three
GT8 Cludlcnnes
Chillenne
Glamoed
239 238 1 24.5 375 Club Maroochy 00000072 250
225 25 23 350 Nambour 1 0 50 72 -22 3.5 1 240 256
-16 16 250 Tewantin-Nossa 0 42 66 -24 4.5 0 197
273 -76 13.5 225 Buderim 2 66 42 24 7 218 229
-11 11 150 TOP 3 TEAMS FROM EACH SECTION
+ THE NEXT TWO HIGHEST TEAMS ON POINTS,
QUALIFY FOR FINALS
Question Answers
Q: How many teams qualify for the finals?
The top teams from each section, plus the next two highest teams on points, qualify the finals, making total of 5 teams.
Q: How many points has the Buderim team earned?
A: The Buderim team has carned 150 points.
Q: Which team is currently in the lead?
A: The Albert team currently the lead with points and a margin 39.
Q: What is the current margin of the Mooloolaba team?
A: The Mooloolaba team has current margin of 1.
Q: What is the overall summary of the document?
The document appears to standings table for the Ladies Division Three of bowling lawn bowls competition, showing the current standings various teams, including their win-loss records, points, and margins.
Pipeline
Image
OCR
LLM
VQAs
Visual Question Answering pairs and the corresponding pipeline. The OCR text is extracted from the image, and passed to the LLM (i.e., Llama 3.3 70B) to generate QA pairs.
Figure 18:
49

---

OCR
Lw [Lg]
10"
10"
Froction of onjects
0.2
SB 1011 10 Fraction of objects 0.2 0.1 0.0 0.01 0.1 1.0
10.0 erg
0.1
0.0
(300
Detailed Caption
The image depicts a histogram of the distribution of objects, with the x-axis labeled "Lsel "r0*"crgs" crgs 'I" and the y-axis labeled "Fraction of objects." The x-axis ranges from 0.01 to 10.0. while the y-axis ranges from 0.0 to 0.2. The histogram is divided into bins of varying widths, with the first bin spanning from 0.01 0.1, the second Each bin contains
bin spanning from 0.1 1.0, and range. The bars are shaded gray, with some bins having multiple bars. A key feature the histogram is presence of a peak in the middle bin, which corresponds to an LsB value of around 1.0. This suggests that the majority of objects have LsB value close to this value. The histogram also shows a tail extending towards higher Lsn values, indicating that objects have significantly higher Lse values than the majority. Overall, the histogram provides visual representation the distribution Ls g values among objects being studied. allows for casy identification of patterns and trends the data, such as the peak in the middle bin and the tail towards higher LsB values.
representing the fraction of objects within that
Multi-Choice Question Answer (MCQA)
What is the approximate LsB value which the majority of objects have their peak?
Options:
(A) 0.1
(B) 1.0
(.0 5.0
(D) 10.0
Answer: (B) 1.0.
Pipeline
Detailed Captions
Image
OCR
MCQAs
Captioner
Figure 19: Detailed Captions and Multi-Choies Question Answers (MCQAs) and the corresponding pipeline. The OCR text is extracted from the image, and the caption is generated by the captioner (i.e., Llama 3V 90B), which all passed to the LLM (i.e., Llama 3.3 70B) to generate MCQAs.
50

---

Metadata
Title: Lions VS Colts Highlights 2017 Preseason Game
Description: Comment suggestions for future videos and Enjoy!
Frame Caption
Frame 435: The image shows a man with dreadlocks standing in front of crowd of people in stadium. He is wearing a white t-shirt and is surrounded by a group of people standing on the ground. On the left side of the image, there is a table fan, bottles, and other objects placed on table. In the background, there are people sitting on chairs, stairs, railings, boards with text, lights, and the sky. The text the boards reads "Indianapolis Colts San Francisco 49crs
Frame 585; The image shows a football game being played on a TV sereen, with group of people on the ground and a few people standing in the background. At the bottom of the image, there is text and numbers indicating that the game is between the Indianapolis Colts and the Detroit Lions.
Frame 765: The image shows group of people playing a game of football on a green field, with white lines marking the boundaries of the field. At the bottom of the image, there is text and numbers indicating that the game is between the Indianapolis Colts and the Detroit Lions.
Frame 945; The image shows football game being played on TV screen, with people wearing helmets and playing on the ground. At the bottom of the image, there text and numbers indicating that the game is between the Detroit Lions and the Indianapolis Colts.
Video Caption
Football players wearing helmets, Detroit Lions vs Indianapolis Colts, player running with ball, falls down, touchdown scored.
Detailed Caption
A football player is running with the ball and then falls down, the game is between the Detroit Lions and the Indianapolis Colts, with players wearing blue and white uniforms and helmets, and people sitting in the stadium, a watermark the image shows the teams playing, one player is laying on the ground next another player wearing a white and blue jersey and white helmet, a horseshoe-shane -sapee object is on the grass, text on the image indicates a touchdown, players are wearing helmets and playing on the ground, with watermarks in the bottom left corner of the images, the game involves the Detroit Lions and the Indianapolis Colts, and also shows the Colts playing against the New England Patriots and the Lions playing against the Dallas Cowboys.
Multi-Choice Question Answer (MCQA)
What is the outcome of the football player's action?
Options:
(A) the player runs out of bounds
(B) the player scores touchdown
(C) the player drops the ball
(D) the player falls down
(E) the player gets injured
Answer: (B) the player scores touchdown.
Pipeline
Video
Captioner
Video Caption
LLM
Detailed Captions
LLM
MCQAs
Captioner
Frame
Captions
Metadata
Figure 20: Detailed Captions and Multi-Choice Question Answers (MCQAs) and the corresponding pipeline. An LLM (i.e., Llama 3.3 70B) with video caption, frame captions fps, and additional metadata (e.g., video title etc.), generates a detailed caption and multiple-choiees question answers pair.
51

---

Qualitative Results
Question: Provide detailed description of the image.
The image is pink postage stamp featuring map of the Grenadines of St. Vincent, with red border and black text. The map shows various locations on the island, including hotels, bays, other points of interest. "Key Features:
Answer:
GRENADINES St. VINCENT
PONN
*Map:* pink map of Grenadines St. Vincent, with border and black text.
BULLNND
*Locations:* Various locations on the island are marked, including hotels, bays, and other points of interest.
MACARN bar
PALTTRE
*Scale:* A scale in miles is provided the bottom of the map.
*Denomination:" The denomination "30c" printed in bottom-right
GRLCTALL
corner of the stamp.
Mustique Island
"Border:* The stamp has red border with a serrated edge.
"Overall Impression:"
30
The image appears to be vintage postage stamp from the Grenadines of St. Vincent, featuring map of the island and various locations of interest. The stamp has classic design and is likely from the mid-20th century.
Figure 21: PLM-8B OCR examples.
Question: What does the word say together from the bikers?
The word says In- dianapolis.
Answer:
Figure 22: PLM-8B OCR in the wild examples.
Question: How tall is the kitty character in the image? Answer: The kitty charac- ter is approximately 100 cm tall.
Figure 23: PLM-8B OCR in the wild examples.
52

---

Question: What is inside the oven?
Answer: There a baking sheet inside the oven.
Question: What tempera-
the oven to?
Answer: The oven set to 200 degrees.
Question: What unit is the oven's temperature to? Answer: The oven's tem- perature set to Celsius.
Figure 24: PLM-8B Natural image QA
Question: Describe the im- age in details
Answer: A city skyline with large body of water in the foreground. The city skyline is made up of tall buildings, some of which skyscrapers. The build- ings densely packed gether and vary in height and color. In the foreground, there is a large body of wa- that appears to be har- bor or bay. There are many boats and ships in the water, some of which are docked at the shore while others are sailing anchored in the middle of water.
Figure 25: PLM-8B Captioning.
53

---

RDCap
Question: Describe the subject's 's actions within the rod rectangles, providing start and end frames ids and brief description for each action in dense caption format. For cach item use the format [start, end]; [description] separated by newline, where Answer: Frame [O, 6]: Out of frame
and end are frame numbers between 0 and 31 in this frame video.
Frame [6, 15]: A woman is walking towards the camera with a child. She is holding the child's hand with her right hand and carrying bag in her left hand Frame [16, 31]: Out of frame.
RCap
Question: What is happening the subject the region highlighted by the red rectangle during frames [6, 15] this 32 frame video?
Answer: a woman is walking on muddy road with a child. she is holding the child's hand with her right hand and adjusting her saree with her left hand while walking
RTLoc
frame of when 'an individual standing near a water body is drinking a from a water from bottle' happens correspondigg to the region marked by the blue rectangle. Use the format [start, end] where start and end are frame numbers between and 31 in this 32 frame video. Answer: [23, 26]
Question:
Please provide start
Captioning
Question: Please describe the video in detail, focusing actions.
Answer: This video features group of people walking around a muddy area with small bridge background. The video is taken outdoors and the weather is sunny. The video starts with a man standing on the left side of the frame, facing the camera. He is wearing a white shirt, black vest, black trousers and white shoes. He has scarf around his neck. The camera pans the right and shows group of people walking around. The video taken from a low angle and the camera handheld.
The vidco ends with a man walking towards the left side of the frame. He is wearing a blue shirt, black trousers and brown shoes. He is holding bottle in his right hand and is drinking from it.
Fine-grained QA
Question: Please explain what the man in blue t-shirt pointing the video. Answer: The man in blue t-shirt is pointing at the bridge.
Figure 26: PLM-8B Video Region Captioning, Grounding and Fine-grained QA.
54

---

Question: Please give me a details of the vidco focusing on actions and interactions.
Answer:
This video features a close shot of a hand holding plush toy mouse with grey kitten standing white chair, reaching out and touching the toy with its paws. In the beginning of the video, the kitten seen standing
the chair and reaching withdraws its paws and sits back on the chair. closer to the kitten and then away from the kitten. The kitten then reaches out again and touches the toy with its paws. The hand then moves the toy away from the kitten again and the kitten looks to the left of the frame.
with both paws touch the toy mouse. The kitten then
hand holding the toy mouse seen moving the toy
This vidco is against white background.
Figure 27: PLM-8B Video Captioning
Question: What are the ingredients being thrown into the mixer? Answer: Carrots, apples, and bananas.
Figure 28: PLM-8B Video QA
55

---

K Limitations and Future Work
Our PLM models achieve strong performance against open-data baselines and proprietary models alike, however there is still room for improvement in both modeling and data. On the model front, we do not experiment extensively with long video modeling components (e.g., token compression, dynamic temporal resolution). As result, our performance on long video benchmarks [41, 123, 125] less competitive (see Table E). PLM is compatible with such newer advancements and can be incorporated in future work.
Additionally, our results sensitive the characteristics of the base LLM. We see especially low performance of PLM on benchmarks such as MMMU [68], MME [95] and Video-MME [20] (see Tables 3 and 4), where the strongest baselines often rely on LLMs that are more verbose, but also have a likely much larger language component (see the gap to proprietary models on some benchmarks). We also note that our model performs relatively poorly on our SGQA task (Table 5), targeting a mix of perception and knowledge based questions to smart glasses. Strong GPT-40 excel at tasks that go beyond core perception.
systems like
On the data front, our mix focuses squarely on visual perception - it does not include for example, multi-step reasoning, roboties world-knowledg data. Despite these limitations, PLM contributes new capabilities and strong benchmark results, and set a new standard for fully reproducible VI.Ms.
L Broader Impact
Our work aims advance open and reproducible research in vision-langusen modeling by releasing models, data, and benchmarks that support open research. By not having any distillation from proprietary models, we hope improve reproducible and transparent training and evaluation of VLM research. However, like all MILMs, our Perception Language Model (PLM) may have some risks. Even by carefully selecting datasets and apply several mitigation (CSAM, NSFW, etc.), the model may still contain hidden biases generate inappropriate or harmful content. We took steps to reduce these risks by teaching the model to refuse answering questions related to bias, harassment, or adult content. We also remove all samples containing any mention of human faces from all the datasets.
We also annotate and release large-scale dataset for fine- ame-grained video question answering and spatio-tempont grounding. This release has the potential to significantly advance research in image and video understanding. Making the dataset openly available allows others to reproduce our work and invites broader community involvement. This transparency supports safer and more accountable progress, helping researchers better understand and address potential biases or limitations.
We believe that by openly sharing our models and data, while actively addressing ethical concerns, work can contribute positively vision-language research.
56

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer Section 3.2 Scaling Law with Synthetic Data and Experiments in Tables 4, 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section K about Limitations and Future Work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
57

---

Justification:
This work does not have any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Experiments Section 5 and training details in Section A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
58

---

Answer: [No]
Justification: will release training and evaluation codes, checkpoints, and all the dataset described in Section 2 and Appendix A upon de-anonymization. However, cannot provide code for submission due to the difficulty of anonymizing it, or the newly annotated datasets due to the size limit.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
public/gan gatlcc/gu///////e (CodeSumissionn
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wwtppccpppcccccccccccccccccccttt /ttee//uuuddesoeeuuumuiii SordeSuumissiin ssisssoliccyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity. the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer Section 5 and Appendix A. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the cost of training large models, repeating most of the experiments in the paper would be prohibitively expensive. Thus, we do not provide error bars. However, we attempt to be thorough as possible by testing on wide variety of over 50 image, multi-image and video benchmarks (refer to Section 5).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
59

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етrоr rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Given the breadth of experiments presented, it is hard detail the exact compute resources for each experiment. Instead, we provide all hyperparameters and data splits in Appendix A to facilitate reproducibility. Additionally, will release the training code and datasets publicly, enabling users to estimate the training FLOPS utilization.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: btttps://neurppe httpppsccccpppcccepeeeeeeeeseeee htte//tthhiiiiieeeeseeeeeeeeeaae ines?
Answer: [Yes]
Justification: We follow the ethical guidelines: annotators to local laws and regulations, the datasets we will release will be properly licensed and conform legal and privacy agreements. In addition, datasets involving humans were properly anonymized using face blur, in accordance with the NeurIPS Code of Ethics. Guidelines:
compensated according
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Section L..
Guidelines:
60

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please refer to Section L.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In this work, openly available dataset and model and properly cited them adhering to the original release license. Please refer to Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
61

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/coo/////ooooo comddataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
While the dataset and models will be publicly released along with licensing terms and training specifications, they are not part of this submission due to space constraints. As a result, we mark this section as not applicable.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Please refer to Appendix F and Appendix G.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Please refer Appendix F and Appendix G.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
62

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: LLMs are used to generate data for our data engine, and the usage LLMs is thoroughly documented for that purpose (e.g. in Section 3.1 and Appendix F, G). Further, our PLMM build on top of openly available LLaMA pretrained LLM (refer to Section 2).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http://eeuur.peeeee cc/Conferencees22255 for what should or should not be described.
63


Reviews:
[{"review_id": "6VqzNP2qiG", "reviewer": "Reviewer_yuJr", "summary": "This paper constructs a fully open and reproducible Perception Language Model (PLM) and releases a large-scale, high-quality, human-annotated video question-answering and spatio-temporally grounded video captioning dataset. Simultaneously, the paper introduces a new PLM-VideoBench benchmark, aiming to promote transparent research in image and video fine-grained understanding and address the limitations of existing synthetic data in handling complex tasks.", "questions": "1. I am curious how the model learns the ability for temporal localization (obtaining action frame indices) in RTLoc and RDCap. Apart from the training data and the positional encoding design in the Perception Encoder, are there any other factors that contribute to the model's temporal localization capability?\n\n2. Can the PLM-VideoBench benchmark effectively evaluate the model's \"long-term dependency\" and \"complex interaction\" reasoning abilities? For example, are there specific questions designed to test the model's ability to understand the same event or person across multiple (non-contiguous) video segments?\n3. Regarding Table 6's ablation study: Is the only varying parameter the data ratio, meaning the total SFT data, including PLM data, is also capped at 4M? If not, and the experiment states that the additional introduction of PLM data brings performance gains, it would weaken the effectiveness of PLM-built data.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. **Complete Data Construction Plan**: The paper provides a complete and comprehensive solution for constructing VLM image/video understanding data based on open-source information. This solution thoroughly discusses the advantages and disadvantages of synthetic and human-annotated data, enlightening practitioners on how to purposefully build visual understanding data.\n\n2. **Adapted Open-Source Training Strategy**: The paper offers a set of training strategies adapted to the aforementioned data plan and based on open-source solutions. This training strategy achieves performance comparable to open-weight models without the limitations of distilling from proprietary models, pushing the performance ceiling for fully open models.\n\n3. **More Challenging Video Benchmark**: The paper introduces a more challenging video benchmark, PLM-VideoBench. This benchmark is constructed based on human annotation, enhancing the evaluation of fine-grained and spatio-temporally grounded video understanding, thus complementing existing video benchmarks.\n\n**Weaknesses**\n\n1. **Insufficient Discussion on Resource Consumption**: The paper emphasizes reproducibility as a contribution but does not detail the specific computational resources required for training and deploying PLM (e.g., number of GPUs, training duration, total FLOPs). \n\n2. **Architectural Generality Rather Than Innovation**: The model architecture (vision encoder + MLP projector + language decoder) is quite common in current VLM research. While the paper's focus is on openness, data, and training recipes, it lacks significant innovation at the model architecture level.", "was_revised": true, "final_justification": "My concerns have been well addressed; I raise my score to Strong Accept.", "rebuttal": "We thank the reviewer for recognizing the contributions of our work. In particular, the acknowledgment of the fully open and reproducible Perception Language Model, the release of large-scale, high-quality human-annotated video QA and grounded captioning data, and the introduction of the PLM-VideoBench benchmark for fine-grained video understanding.\n\nBelow, we provide answers to the reviewer's questions. \n\n---\n> 1. **Insufficient Discussion on Resource Consumption**: The paper emphasizes reproducibility as a contribution but does not detail the specific computational resources required for training and deploying PLM (e.g., number of GPUs, training duration, total FLOPs).\n\nThe full training process for the PLM-3B model, including all three stages, takes approximately 5.5 days on 128 H100 GPUs. The PLM-1B model requires less training time, while the PLM-8B model takes slightly longer. The detailed breakdown is listed in the table below. \n\n| Model              | Stage 1 (GPUs & Time) | Stage 2 (GPUs & Time) | Stage 3 (GPUs & Time) |\n|-------------------|------------------------|------------------------|------------------------|\n| PLM-1B (PE L/14)  | 8 GPUs & 3 hours       | 128 GPUs & 1.5 Days    | 128 GPUs & 2.0 Days    |\n| PLM-3B (PE L/14)  | 8 GPUs & 4 hours       | 128 GPUs & 3.0 Days    | 128 GPUs & 2.5 Days    |\n| PLM-8B (PE G/14)  | 8 GPUs & 6 hours       | 256 GPUs & 3.0 Days    | 256 GPUs & 2.5 Days    |\n\nFurther, at inference time, all models can run efficiently on a single GPU: the 1B model on a 24GB GPU, the 3B model on a 40GB GPU, and the 8B model on an 80GB GPU. We will include these details in the final version to support reproducibility.\n\n---\n\n> 2. **Architectural Generality Rather Than Innovation**: The model architecture (vision encoder + MLP projector + language decoder) is quite common in current VLM research. While the paper's focus is on openness, data, and training recipes, it lacks significant innovation at the model architecture level.\n\nWe agree with the reviewer that the model architecture follows commonly used components (vision encoder + MLP projector + language decoder) and architectural innovation is not the focus of this work. Instead, we focus on developing a reproducible recipe for a broad spectrum of VLM models. Therefore, we chose to keep the architecture simple with the most popular choice of VLM components, as also in LLaVA [1], QwenVL [2], Molmo [3] and InternVL [4].\n\nOur primary contributions lie in building a fully open and reproducible pipeline, including a scalable synthetic data engine, analysis of scaling behavior across data modalities, and identifying critical gaps in existing training data. To address these, we introduce the PLM-FGQA and PLM-STC (our human annotated datasets) and the PLM-VideoBench benchmark, along with an open-source training recipe. Together, these components result in a fully open model that achieves state-of-the-art performance across 40+ benchmarks.\n\n---\n[1] Visual instruction tuning [Liu et al., arXiv 2023]\n\n[2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond [Bai et al., arXiv 2023]\n\n[3] Molmo and Pixmo: Open Weights and Open Data for State-of-the-Art Multimodal Models [Deitke et al., arXiv 2024]\n\n[4] Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks [Chen et al., arXiv 2024]\n\n---\n\n> 1. I am curious how the model learns the ability for temporal localization (obtaining action frame indices) in RTLoc and RDCap. Apart from the training data and the positional encoding design in the Perception Encoder, are there any other factors that contribute to the model's temporal localization capability?\n\nIn addition to the training data and the temporal positional encoding in the Perception Encoder [1], the way we prompt the model also plays a key role in enabling temporal localization. Specifically, we include the number of sampled frames in the video as part of the input prompt, which helps the model frame the task as predicting the start and end frame indices within a known temporal window. We find that this explicit prompting improves the model's ability to learn temporal boundaries in tasks such as RTLoc and RDCap.\n\n---\n[1] Perception Encoder: The best visual embeddings are not at the output of the network [Bolya et al., arXiv 2025]\n\n---\n\n> 2. Can the PLM-VideoBench benchmark effectively evaluate the model's \"long-term dependency\" and \"complex interaction\" reasoning abilities? For example, are there specific questions designed to test the model's ability to understand the same event or person across multiple (non-contiguous) video segments?\n\nYes, PLM-VideoBench is designed to evaluate long-term dependency and complex interaction reasoning abilities. In particular, the RDCap task includes annotations for each subject in the form of multiple non-contiguous and non-overlapping (start, end, action-focused caption) triplets. These triplets describe distinct temporally grounded actions performed by the same subject across different segments of the video. Further, RDCap explicitly provides temporal boundaries where the subject is not visible in the frame. During evaluation, the model is expected to generate these multiple event descriptions grounded in time, which directly tests its ability to track entities and reason over extended temporal contexts.\n\n---\n\n> 3. Regarding Table 6's ablation study: Is the only varying parameter the data ratio, meaning the total SFT data, including PLM data, is also capped at 4M? If not, and the experiment states that the additional introduction of PLM data brings performance gains, it would weaken the effectiveness of PLM-built data.\n\nThank you for your insightful question regarding the ablation study in Table 6. We would like to argue that controlling the number of total data samples for each ablation blend introduces greater confounding factors, since our SFT blend (stage 3) is a huge collection from different data sources. Instead, we maintain the standard practice of training all settings using 1 epoch in total, ensuring a consistent evaluation framework."}, {"review_id": "Vwr4aMgglZ", "reviewer": "Reviewer_dG2e", "summary": "The paper introduces the Perception Language Model (PLM), an open and fully reproducible model aimed at advancing visual and video understanding. The authors address the limitations of proprietary vision-language models by providing transparent training processes, including both synthetic and human-annotated data. They also introduce PLM-VideoBench, a new benchmark suite focused on fine-grained video understanding and spatio-temporal reasoning. Through this work, the authors aim to enhance reproducibility in VLM research and to bridge gaps in detailed video understanding, especially in tasks requiring fine-grained temporal and spatial reasoning.", "questions": "Can the author provide a plot for the distribution of synthetic data and real annotated data. This may be helpful to analyze the limitation of synthetic data in section 3.", "limitations": "The author can add potential negative societal impact, especially training with unverified synthetic data.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1) The authors emphasize transparency and reproducibility, offering complete datasets, training recipes, and models to facilitate further research in visual understanding.\n    \n2)  The release of new 2.8 M human-labeled instances of fine-grained video question-answer pairs and spatio-temporally grounded captions significantly enhances the data available for training VLMs, addressing existing gaps in video understanding tasks.\n    \n3) PLM performs competitively against existing state-of-the-art models, including proprietary models, across multiple image and video benchmarks, showcasing its efficacy despite the use of only open-access and synthetic data.\n    \n4) The introduction of PLM-VideoBench is a valuable contribution for evaluating challenging video understanding tasks that involve spatial, temporal, and fine-grained activity reasoning.\n\n**Weakness**\n\nA deeper analysis of how the syntheic/realistic quality and proportion of synthetic data contribute to video reasoning could be valuable to the community.", "was_revised": true, "final_justification": "After reviewing your response and the other reviewers’ comments, I’m maintaining my recommendation for acceptance.", "rebuttal": "We thank the reviewer for the acknowledgment of our efforts toward transparency and reproducibility, including the release of 2.8M human-labeled video question-answer pairs and grounded captions. We also value the recognition of PLM’s competitive performance across image and video benchmarks, against both open-source and proprietary models, as well as the introduction of PLM-VideoBench for fine-grained spatio-temporal evaluation.\n\n---\n\n> A deeper analysis of how the synthetic/realistic quality and proportion of synthetic data contribute to video reasoning could be valuable to the community.\n\nWe thank the reviewer for this suggestion. A deeper analysis of synthetic data quality and proportion is indeed valuable, and we provide relevant results in Appendix B. Specifically, we show synthetic data scaling laws for video reasoning tasks such as EgoSchema, MVBench, VideoMME, and PerceptionTest, analyzing model performance with respect to both vision encoder size (Figure 7) and input size (Figure 8). Further, dataset-specific trends are available in Figure 9. We hope these analyses offer useful insights into the role of synthetic data in video understanding.\n\n---\n\n> Can the author provide a plot for the distribution of synthetic data and real annotated data. This may be helpful to analyze the limitation of synthetic data in section 3.\n\nThe distribution of synthetic and human-annotated data used in training is detailed in Table 9 of Appendix A. This table breaks down the number of samples from each dataset, covering both synthetic and manually annotated data across image and video tasks. While we are currently unable to include plots in the author response due to NeurIPS policy, we will include a visual representation of this distribution in the final version of the paper.\n\n---\n\n> Limitations:\n> The author can add potential negative societal impact, especially training with unverified synthetic data.\n\nWe acknowledge that training with large-scale synthetic data may carry risks related to data quality and unintended biases. In our Appendix M (L1636), we have discussed the broader impact of our project. Specifically, our dataset has undergone a number of careful data mitigation (CSAM, NSFW, etc.), as well as additional filtering of harmful content and human subject and face-blurring."}, {"review_id": "Eo435N3BNr", "reviewer": "Reviewer_s99d", "summary": "The main contributions of this work are embodied in its transparent multimodal data construction pipeline, which incorporates both synthetic data and manually annotated data. By exploring scaling laws with synthetic data, this study reveals that even extensive synthetic datasets struggle to improve Vision-Language Models' performance on challenging tasks and novel scenarios, thereby underscoring the critical importance of high-quality human-annotated data.\n\nThe contributions of this work include:\n1. Transparent synthetic data pipelines\n2. Open-access datasets and models\n3. Enhanced reproducibility supported by comprehensive experimental analysis and systematic interpretation.", "questions": "1. Does the paper provide a clear explanation regarding \"the significance of each training stage\"?\n2. Will other synthetic data (66.1M) besides the high-quality human-annotated video data also be open-sourced?", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The transparent synthetic data construction pipeline, open-access high-quality human-annotated data, and transparent training strategies are highly valuable for the VLM research community.\n2. PLM–VideoBench addresses a critical gap in the research community by providing a benchmark for evaluating VLMs' holistic video understanding capabilities.\n3. Extensive experiments demonstrate that these transparent pipelines and datasets enable reproducible performance comparable to both proprietary models and open-weight models.\n4. The response regarding \"data gaps\" is sufficiently addressed.\n\nWeaknesses:\n\nMy only minor concern pertains to the discussion of \"the significance of each training stage\" mentioned in line 30: While I can find some relevant explanations in lines 73-92 and Appendix A.1, these sections primarily focus on implementation details and training settings. Therefore, it remains unclear to me what \"the significance of each training stage\" specifically refers to, and the authors' systematic exposition of this question appears insufficient.", "was_revised": true, "final_justification": "All concerns have been addressed, and I am willing to raise my score.", "rebuttal": "We thank the reviewer for acknowledging the main contributions of our work, including the transparent synthetic data pipeline, open-access human-annotated datasets, and the comprehensive training and evaluation framework. We appreciate the recognition of PLM-VideoBench as a valuable benchmark for holistic video understanding, as well as the usefulness of our systematic analysis in identifying and addressing data gaps in existing VLM training datasets.\n\n---\n\n> 1. My only minor concern pertains to the discussion of \"the significance of each training stage\" mentioned in line 30: While I can find some relevant explanations in lines 73-92 and Appendix A.1, these sections primarily focus on implementation details and training settings. Therefore, it remains unclear to me what \"the significance of each training stage\" specifically refers to, and the authors' systematic exposition of this question appears insufficient.\n\n> 1. Does the paper provide a clear explanation regarding \"the significance of each training stage\"?\n\nWe thank the reviewer for pointing out the need for a clearer explanation of the significance of each training stage. Table 6 provides a controlled comparison in its first two rows that shows the impact of Stage 2. In the first row, only stage 1 + stage 3 is performed, serving as a baseline. In the second row, synthetic data (PLM-Synth ✓) is included, corresponding to Stage 2 of our training pipeline. Importantly, PLM-STC and PLM-FGQA are not included in either of these two configurations. This setup shows that introducing Stage 2 leads to consistent improvements across nearly all metrics.\n\n Stage 2 | Total Avg | PLM-FGQA (MBAcc) | PLM-SGQA (acc) | PLM-ST (3-metric avg.) | Fine-Grained QA | Video Cap (Dream 1K) | Video QA | Video Hallu | Spatio & Temp | Image OCR | Image Cap | Image Rec |\n| --------- | --------- | ---------------- | --------------- | ---------------------- | --------------- | -------------------- | -------- | ----------- | ------------- | --------- | --------- | --------- |\n| ✗         | 48.5      | 39.7             | 34.4            | 6.6                    | 42.2            | 24.0                 | 67.5     | 64.9        | 50.6          | 76.0      | 64.3      | 63.3      |\n| **✓**     | **54.3**  | **49.8**         | **35.9**        | **14.7**               | **48.8**        | **29.9**             | **73.2** | **73.3**    | **56.1**      | **84.0**  | **65.9**  | **65.5**  |\n\n---\n\nIn addition to Table 6, we conducted an ablation study where the stage 2 and 3 are combined. Specifically, we run PLM 3B with a combined data blend (total 90M+) at 36 max number of tiles and 32 max number of frames setting (i.e., stage 3 setup) and compare with PLM 3B below: \n\n### **Image Benchmarks**\n|              |   AVG        |   DocVQA     |   ChartQA    |   TextVQA    |   Info.QA   |   AI2D       |   OCRBench   |   MMMU       |   VQAv2      |   OK-VQA     |   VizWiz   |   SEED       |   BLINK      |   CVBench    |   RealworldQA   |   VSR        |   POPE       |\n| :----------: | :----------: | :----------: | :----------: | :----------: | :----------------------------: | :----------: | :----------: | :----------: | :----------: | :----------: | :--------: | :----------: | :----------: | :----------: | :-------------: | :----------: | :----------: |\n|   Stage2+3 (combined)   |   75.3       |   91.8       |   82.6       |   **84.5**   |   71.2                         |   90.0         |   **83.7**   |   39.9       |   83.6       |   **67.4**   |   62.8     |   78.5       |   54.6       |   72.5       |   **72.8**      |   79.9       |   **89.6**   |\n|   PLM-3B      |   **76.5**   |   **93.8**   |   **84.3**   |   84.3       |   **74.6**                     |   **90.9**   |   83.0         |   **41.2**   |   **84.3**   |   66.8       |   **64**   |   **78.5**   |   **55.4**   |   **81.4**   |   72.4          |   **80.4**   |   88.7       |\n\n\n\n### **Video benchmarks**\n|     | AVG | MVBench | NextQA | Percept. Test | STAR | VideoMME | ActivityNetQA | EgoSchema (val) | Temp. Bench | TOMATO | MotionBench | TempCompass | CG-Bench | Charades| VideoHall. | EventHall. |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Stage2+3 (combined) | 60.5 | 71.6 | **84.4** | 73.9 | 82.9 | **56.7** | **67.5** | 71.8 | **25.7** | 28.5 | 57.2 | **69.8** | 44.9 | 47 | 53.4 | 71.9 |\n| PLM3B | **62.5** | 74.7 | 83.4 | **79.3** | **84.8** | 54.9 | 66.2 | **72.6** | 23.4 | **30.9** | **60.4** | 69.3 | **47.2** | **57.7** | **55.5** | **76.5** |\n\nDespite nearly doubling the total training FLOPs, the combined setup results in overall worse performance compared to our three-stage training pipeline (on average **1.2** image, **2.0** video). This highlights the importance of the staged design where:\n- Stage 1 warms up the projector (MLP) with small image-caption data.\n- Stage 2 enables broad scaling using synthetic image and video data at moderate input resolution.\n- Stage 3 fine-tunes the model on high-quality human-labeled data for high input resolution.\n\nThis progressive approach helps stabilize training and distribute training FLOPs efficiently. We note that training MLLMs in multiple stages is now becoming a standard practice as seen in previous works such as LLaVA-OneVision [1], Qwen-VL [2], Molmo [3], and InternVL-2 [4], etc.\n\n---\n\n[1] LLaVA-OneVision: Easy Visual Task Transfer [Li et al., arXiv 2024]\n\n[2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond [Bai et al., arXiv 2023]\n\n[3] Molmo and Pixmo: Open Weights and Open Data for State-of-the-Art Multimodal Models [Deitke et al., arXiv 2024]\n\n[4] Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling [Chen et al., arXiv 2025]\n\n---\n\n> 2. Will other synthetic data (66.1M) besides the high-quality human-annotated video data also be open-sourced?\n\nYes, all datasets used in this work, including both the 66.1M synthetic data and the 2.8M human-annotated video data, will be open-sourced to support transparency and reproducibility in future research."}, {"review_id": "jdsy6upOh1", "reviewer": "Reviewer_e1P7", "summary": "This paper addresses the issue of non-transparency in proprietary vision-language models such as GPT-4o, which makes it difficult for the academic community to accurately assess the true capabilities of current models. To overcome this limitation, the authors propose a data engine designed to generate synthetic multimodal data in a scalable yet transparent manner. Using the generated data, they conduct a series of scaling experiments and observe that while synthetic data is effective for general video understanding, it falls short on more challenging video comprehension tasks. To mitigate this, the paper introduces a high-quality video dataset annotated by humans, along with PLM-VideoBench, a benchmark derived from the human-annotated dataset. With a carefully designed training pipeline and data engine, the resulting model family - PLMs - achieves strong performance across a wide range of image and video benchmarks.", "questions": "1. Will the data engine be open-sourced or not?\n2. Can the limitation that synthetic data does not significantly improve performance on hard video QA tasks be mitigated by using a stronger VLM or LLM?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. This paper is well-motivated. Currently, the widespread use of proprietary models as data annotators makes it difficult to study the influence of other contributing factors. A fully transparent and controllable synthetic data engine can significantly benefit research in this area.\n2. The dataset and benchmark contributions are both substantial and solid. The inclusion of both synthetic data and human-annotated data provides valuable resources for advancing VLM development.\n3. The paper conducts extensive experiments on the scaling behavior of synthetic data and finds that while synthetic data benefits general video understanding, it is less effective for more challenging video QA tasks.\n\n### Weakness\n\n1. Although the paper emphasizes transparency, it adopts LLaMA3 as an open-source alternative to closed-source models. However, LLaMA3’s training process lacks full transparency, and it may have been trained on content generated by proprietary models. This could implicitly influence the quality of the synthetic data and affect PLM performance.\n2. Regarding model performance on image benchmarks, PLMs exhibit strong OCR capabilities but perform poorly on knowledge-intensive tasks such as MMMU. This raises the question of whether such disparities stem from biases in the data engine.\n3. A similar pattern is observed in video benchmarks: PLMs perform well on PerceptionTest but fall significantly behind on VideoMME. This inconsistency warrants further investigation.\n4. Since PLM-VideoBench is an in-distribution byproduct of the high-quality human-annotated dataset, the model’s performance advantage on this benchmark over other models is less compelling.", "was_revised": true, "final_justification": "The rebuttal resolves most of my concerns; accordingly, I maintain my original rating and recommend acceptance.", "rebuttal": "We thank the reviewer for recognizing the key contributions of our work. We appreciate the acknowledgment of the motivation behind addressing the non-transparency of proprietary vision-language models such as GPT-4o. We also appreciate the recognition of our transparent and controllable synthetic data engine, the release of both synthetic and human-annotated datasets, and the scaling analysis that highlights the limitations of synthetic data on challenging video reasoning tasks.\n\n---\n\n> 1. Although the paper emphasizes transparency, it adopts LLaMA3 as an open-source alternative to closed-source models. However, LLaMA3’s training process lacks full transparency, and it may have been trained on content generated by proprietary models. This could implicitly influence the quality of the synthetic data and affect PLM performance.\n\nWe thank the reviewer for the observation. In our work, we differentiate between closed-weight proprietary models (e.g., GPT-4, Gemini) and open-weight models such as LLaMA3. While LLaMA3’s full training corpus is not publicly available, its model weights, training setup, and outputs are accessible and reproducible, unlike closed models whose generations cannot be independently verified. We select LLaMA3 specifically to ensure transparency and reproducibility in our synthetic data engine. \n\nWe understand the reviewer’s concern that the full data source of LLaMA training is not disclosed. We have contacted the authors of LLaMA3 and confirmed that no artifacts of proprietary models (e.g., contents generated by closed models) was used in training LLaMA3.\n\n---\n\n> 2. Regarding model performance on image benchmarks, PLMs exhibit strong OCR capabilities but perform poorly on knowledge-intensive tasks such as MMMU. This raises the question of whether such disparities stem from biases in the data engine.\n\n> 3. A similar pattern is observed in video benchmarks: PLMs perform well on PerceptionTest but fall significantly behind on VideoMME. This inconsistency warrants further investigation.\n\nWe thank the reviewer for the observation. We hypothesize that some benchmarks such as MMMU and VideoMME rely heavily on the capabilities of the underlying language model. To test this, we conducted a separate ablation experiment comparing LLaMA3 and other popular LLM. \n\nSpecifically, we extract the vision encoder from PLM after Stage 2, and perform a short warm-up (Stage 1) followed by a shorter version of SFT (Stage 3) using newly initialized LLM, LLaMA3.1-Instruct 8B and Qwen2.5-Instruct 7B models. Beside the LLMs, everything else is controlled (training data, # iterations, random seed, etc.)  \n\nFrom the result, we observe that results are consistently better with Qwen2.5: VideoMME improves from 48.0 to 54.1, and MMMU from 39.9 to 48.1. These results suggest that the disparities on knowledge-intensive tasks are more likely due to differences in LLM capabilities rather than biases in the data engine. We have also noted this in the limitations section (Appendix L, Line 1619) of the paper. \n\n---\n\n> 4. Since PLM-VideoBench is an in-distribution byproduct of the high-quality human-annotated dataset, the model’s performance advantage on this benchmark over other models is less compelling.\n\nWe thank the reviewer for the insightful comment on the comparison of PLM with previous methods where PLM included in-distribution data, and the other methods did not. To mitigate this, we conduct an experiment by training PLM after removing human annotated in-distributed data. The table below shows the results. We notice that PLM still performs better than Qwen2.5-VL and InternVL2.5 models, indicating that PLM performs better in a fair setting where no in-domain data is included in the training. \n\n| Model              | FGQA | SGQA | RDCap | RCap | RTL  |\n|--------------------|------|------|-------|------|------|\n| Qwen2.5 VL 3B      | 43.7 | 45.1 | 0.3   | 17.2 | 13.9 |\n| InternVL2.5 4B     | 50.0 | **49.2** | 4.9   | 25.9 | 15.4 |\n| **PLM 3B no hum.** | **54.0** | 41.5 | **11.1**  | **27.6** | **25.0** |\n| PLM 3B         | 67.1 | 38.8 | 53.1  | 45.0 | 58.2 |\n\n---\n\n> 1. Will the data engine be open-sourced or not?\n\nYes, the data engine will be open-sourced along with the datasets, models, and training recipes to support transparency, reproducibility, and further research in the community. \n\n---\n\n> 2. Can the limitation that synthetic data does not significantly improve performance on hard video QA tasks be mitigated by using a stronger VLM or LLM?\n\nWe thank the reviewer for the insightful question. During the development of PLM, LLaMA3 and LLaMA3V were the best language and vision-language models that were not trained with any content generated by proprietary models. Molmo used Claude for re-captioning, while QwenVLs and InternVLs used GPT-generated data. During development, we tried using LLaMA3.1 405B for our data engine, but the results were similar compared to LLaMA3.3 70B, despite a much higher inference cost. Our data engine was designed after multiple iterations and optimized for both scalability and accuracy.\n\nFurther, regarding the parameter count in PLM, in Figure 3, we show results across three model scales, 1B, 3B, and 8B, and observe the same trend: scaling synthetic data improves performance on general video understanding tasks, but shows limited gains on Hard Video QA tasks. While larger models (e.g., 8B) generally performs better than smaller ones (e.g., 3B), increasing the amount of synthetic data for each does not significantly improve performance on Hard QA (i.e. Cg-Bench, Tomato, EventHallusion, MotionBench, TemporalBench, LVBench, and Charades-STA). These tasks require fine-grained visual understanding over time, and we believe that further increasing the LLM size (e.g., to 70B or even 405B) would still exhibit similar limitations, as the bottleneck lies in visual reasoning rather than language modeling capacity.\n\nHowever, as shown in Figure 5, adding PLM human-annotated data, which provides rich video-level fine-grained QAs and spatio-temporal grounded captions, leads to significant improvements on Hard QA. This supports our view that stronger visual supervision, not larger language models, is key to addressing the limitations in these benchmarks."}]

Author rebuttal:

Meta-review: All reviewers recommend clear acceptance, highlighting the paper’s solid technical contributions, strong empirical validation, and clear impact on the vision-language modeling community. The open-source release further strengthens its significance and potential influence. 

Several concerns raised by the reviewers were adequately addressed in the rebuttal, including:
- Discussion on resource consumption (reviewer `yuJr`)
- Innovation lack at the model architecture level (reviewer `yuJr`)
- Discussion on synthetic data (reviewer `dG2e`, `e1P7`)
- Unclear significance of each training stage (reviewer `s99d`)

Given the uniformly positive reviews and its contributions to both research and practice, I recommend acceptance.
Decision: Accept (spotlight)