LEAP-UQ/video-uq-failure-mode-benchmarks

Video-UQ Failure-Mode Benchmarks

This directory contains controlled MCQ benchmarks for diagnosing four VideoLLM failure modes: language-prior reliance, frame-sampling failure, evidence-extraction failure, and temporal insensitivity. The canonical release contains MVP, KFS-Bench, and the intervention-focused TempCompass counterfactual set. TVBench and the older mixed 300-item temporal subset are not part of the current dataset.

Failure labels are based on greedy answers only. The MCQ prompt shows the available options and asks for the option letter. Any stochastic generations retained from earlier baseline runs are intended for later uncertainty analysis and do not affect failure labels.

Canonical Composition

Failure-mode view	Folder	Source	Evaluation units	Media
Language prior	language_prior/mvp_balanced_300/	Minimal Video Pairs	300 pairs / 600 members	595 videos
Sampling and evidence	sampling_sensitivity/kfs_balanced_300/	KFS-Bench over LongVideoBench and Video-MME	300 questions	300 videos
Temporal grounding	temporal_ordering/tempcompass_counterfactuals_309/	TempCompass	136 groups / 309 members	301 videos

The KFS media folder also contains 121 Video-MME subtitle files. These are auxiliary metadata, not additional videos.

Layout

failure_mode_benchmarks/
  README.md
  language_prior/
    mvp.py
    mvp_balanced_300/
  sampling_sensitivity/
    create_kfs_subset.py
    get_kfs_subset_videolist.py
    extract_kfs_selected_videos.py
    kfs_balanced_300/
  temporal_ordering/
    create_tempcompass_counterfactuals.py
    download_tempcompass_counterfactuals.py
    tempcompass_official/
    tempcompass_counterfactuals_309/

Common Evaluation Process

1. Run each model with the same MCQ prompt and greedy decoding.
2. Resolve the prediction to an option index or canonical option text. Invalid-format responses count as incorrect.
3. Compare answers across controlled videos, visual conditions, frame samplers, or temporal counterfactual members.
4. Assign a failure label only when all interventions required by that label exist.
5. Retain the question, options, reference, generated answer, correctness, sampled frame indices, model, and intervention metadata in every result table.

The five current models are Qwen2.5-VL-7B, LLaVA-NeXT-Video-7B, InternVL3-8B, Qwen3-VL-32B, and InternVL3-38B.

Greedy MCQ Baselines

These figures were recomputed from the canonical video-conditioned Uniform-32 CSVs. The temporal column uses only the new 309-member TempCompass counterfactual set.

Model	MVP, 600 members	KFS, 300 questions	TempCompass, 309 members
Qwen2.5-VL-7B	64.67% (388/600)	55.00% (165/300)	58.58% (181/309)
LLaVA-NeXT-Video-7B	53.00% (318/600)	33.33% (100/300)	19.09% (59/309)
InternVL3-8B	68.50% (411/600)	56.33% (169/300)	64.08% (198/309)
Qwen3-VL-32B	71.33% (428/600)	61.33% (184/300)	73.79% (228/309)
InternVL3-38B	74.33% (446/600)	63.67% (191/300)	71.52% (221/309)

These are member-level accuracies, not failure-label rates. Sampling and evidence labels require oracle comparisons, while language-prior labels require no-video answers.

Language Prior: MVP Balanced 300

Path: language_prior/mvp_balanced_300/

Each MVP pair contains two minimally different videos with the same question and options but different correct answers. The benchmark compares three conditions for each pair and model:

Video A MCQ answer
Video B MCQ answer
No-video MCQ answer

Balance

Configuration	Pairs
human_object_interactions	75
intuitive_physics	75
robot_object_interactions	75
temporal_reasoning	75
Total	300

Expected Labels

Label	Decision rule
grounded_success	Video A and Video B both match their own references.
critical_language_prior	Both videos repeat the same answer as no-video despite the opposing references.
visual_insensitivity	Both videos receive the same answer, but that answer differs from no-video.
partial_grounding	Exactly one video is answered correctly.
ungrounded_visual_sensitivity	Both video answers are wrong but change between Video A and Video B.
equivalence_judge_inconsistent	Answer-equivalence relations are contradictory; exclude from behavioral analysis.

For MCQ labeling, answers are compared by the selected canonical option text rather than by letter alone because option order can differ between rows.

Files

File	Rows	Description
mvp_300_pairs.jsonl / .csv	300	One row per pair, including both videos and pair-specific references.
mvp_300_examples.jsonl / .csv	600	One inference row per individual video.
mvp_300_summary.json	1	Configuration, source, and rejection statistics.
selected_videos/	595	All required videos; five media paths are reused.
mvp_video_map.json	595	Source-relative path to local-media mapping.

Rebuild

cd /home/debarpanb/videouq/failure_mode_benchmarks/language_prior
python mvp.py --total-pairs 300 --seed 42 --output-dir mvp_balanced_300

cd /home/debarpanb/videouq/failure_mode_benchmarks
python download_selected_videos_from_hf.py --suite mvp

Frame Sampling and Evidence Extraction: KFS Balanced 300

Path: sampling_sensitivity/kfs_balanced_300/

KFS provides required evidence scenes and temporal segments. Every practical sampler uses the same model, question, MCQ options, preprocessing, and frame budget.

Balance

Source	Scene group	Duration bins	Count
LongVideoBench	Single scene	25 short, 25 medium, 25 long	75
LongVideoBench	Multiple scenes	25 short, 25 medium, 25 long	75
Video-MME	Single scene	25 short, 25 medium, 25 long	75
Video-MME	Multiple scenes	25 short, 25 medium, 25 long	75
Total			300

There is at most one selected question per underlying video. The annotation table has 951 required evidence segments.

Sampling Process

Condition	Construction
uniform32	Select 32 approximately equally spaced frames.
kmeans32	Cluster one-frame-per-second CLIP image features and select the frame nearest each centroid.
aks32	Use question-frame CLIP similarity and adaptive temporal segmentation to select relevant frames.
random_seed_0/1/2	Select reproducible random frames from the one-frame-per-second candidate pool.
oracle_evidence_32	Allocate exactly 32 frames directly across annotated required scenes and segments.
oracle_evidence_64	Repeat the annotation-guided construction with 64 frames for Oracle-32 failures.

BSR=1 means every required scene receives its required frame allocation. BSR<1 means at least one required scene is missing or underrepresented.

Expected Labels

Label	Decision rule
sampling_failure	Oracle-32 correct, practical sampler wrong, and BSR=1.
evidence_extraction_failure	Oracle-32 correct, practical sampler wrong, and BSR<1.
clean_robust_negative	Practical sampler and Oracle-32 correct with BSR=1.
evidence_missing_but_answered	Practical sampler and Oracle-32 correct with BSR<1.
budget_sensitive_recovery	Oracle-32 wrong and Oracle-64 correct.
oracle_unresolved	Oracle-32 and Oracle-64 both wrong.
oracle32_regression	Practical sampler correct but Oracle-32 wrong.
judge_ambiguous_excluded	A required correctness decision cannot be resolved reliably.

Oracle-64 is a secondary budget diagnostic. It does not replace Oracle-32 when distinguishing same-budget sampling from evidence-extraction failure.

Files

File	Rows	Description
kfs_300_manifest.jsonl / .csv	300	Question, source, video, scene grouping, duration bin, and evidence metadata.
kfs_300_segments.jsonl / .csv	951	One row per required evidence segment.
kfs_300_summary.json	1	Balance and evidence-duration statistics.
video_lists/kfs_300_with_video_refs.jsonl / .csv	300	Manifest with resolved local references.
selected_videos/	300 videos	150 LongVideoBench and 150 Video-MME videos.
selected_videos/video_mme/*.srt	121 subtitles	Auxiliary Video-MME subtitles.

Rebuild

cd /home/debarpanb/videouq/failure_mode_benchmarks/sampling_sensitivity
python create_kfs_subset.py --total 300 --seed 42 --output-dir kfs_balanced_300
python get_kfs_subset_videolist.py
python extract_kfs_selected_videos.py

Temporal Grounding: TempCompass Counterfactuals 309

Path: temporal_ordering/tempcompass_counterfactuals_309/

This set contains complete controlled groups from the official TempCompass processing metadata. Every member in a group uses the same MCQ question, while the video and correct answer change according to the temporal intervention.

Counterfactual Composition

Intervention	Groups	Members	Conditions
Direction reversal	51	102	Original, reversed
Attribute-change reversal	48	96	Original, reversed
Relative speed	18	54	Same speed, slower, faster
Event order	19	57	A before B, B before A, simultaneous
Total	136	309

The 309 members resolve to 301 unique videos. Four reverse-video bases participate in both a direction and an attribute-change question group.

Counterfactual Construction

• Direction and attribute groups pair an official original video with its _reverse version.
• Relative-speed groups use the official _concat_0, _concat_1, and _concat_2 videos and metadata to identify same-speed, slower, and faster conditions.
• Event-order groups use the official concatenations for simultaneous events, event A before B, and event B before A.
• The builder selects one question stem shared by every member and verifies that the authoritative answers distinguish the required conditions.
• Exactly 307 members use an official shared question. Two direction members use one metadata-derived canonical stem because the official wording changes from appear to disappear after reversal.

Expected Labels

Label	Decision rule
temporal_grounded_success	Every member is correct and answers track the counterfactual references.
critical_temporal_insensitivity	Members receive the same answer despite requiring different answers.
partial_temporal_grounding	Some, but not all, members are correct and the answer pattern shows temporal sensitivity.
partial_temporal_insensitivity	Some members are correct, but at least one required counterfactual distinction is collapsed.
ungrounded_temporal_sensitivity	Answers change across members, but none is correct.
temporal_judge_ambiguous_excluded	A required correctness or equivalence decision is genuinely ambiguous.
temporal_judge_inconsistent	Pairwise answer-equivalence decisions are logically inconsistent.

MCQ equivalence compares selected canonical option text, not raw letters. A changed letter alone is not evidence of temporal sensitivity when option order changes.

Files

File	Rows	Description
tempcompass_counterfactuals_309.jsonl / .csv	309	Counterfactual annotations before local path resolution.
tempcompass_counterfactuals_309_final.jsonl	309	Annotations with verified local paths.
temporal_counterfactual_309_manifest.jsonl / .csv	309	Inference-ready MCQ manifest.
required_tempcompass_video_ids.txt	301	Required official video IDs.
tempcompass_video_map.json	301	Video ID to local path mapping.
summary.json	1	Group, member, condition, and reuse counts.
selected_videos/tempcompass/	301	All required videos.

Rebuild

cd /home/debarpanb/videouq/failure_mode_benchmarks/temporal_ordering
python create_tempcompass_counterfactuals.py

source /home/debarpanb/miniconda3/etc/profile.d/conda.sh
conda activate videouq
python download_tempcompass_counterfactuals.py

To regenerate only resolved annotations and manifests from existing videos:

python download_tempcompass_counterfactuals.py --skip-download

Common Fields

• question: natural-language question stem.
• question_with_options: question plus MCQ options.
• candidates or candidates_json: ordered answer options.
• answer or answer_text: authoritative correct option text.
• answer_index: zero-based correct option index.
• answer_letter: corresponding option letter.
• video_path and related video identifiers: required media reference.
• counterfactual_group_id, counterfactual_intervention, and counterfactual_condition: temporal grouping metadata.
• required_segments and required_scene_ids: KFS evidence annotations.

Prefer JSONL for evaluation code and CSV for manual inspection. Summary JSON files provide the quickest integrity check for the intended composition.

TempCompass is distributed for academic research under CC BY-NC 4.0. Preserve the original attribution and license conditions when redistributing this derivative set.