abcdef-higklmn/LongVQU-B4

LongVQU-Eval: team net4u_athena (final Test entry "B4", 62.33%)

MohammadAli Hamidi (University of Cagliari, team leader), Hadi Amirpour (University of Klagenfurt). Codabench handle: net4u_athena.

LoRA adapters (all trained on the full 1,200-question Dev set) for the final system. The base models are not re-uploaded; they load from the Hugging Face Hub.

Folder	Base model	Role
8b_base_exp115	Qwen3-VL-8B-Instruct	base fine-tuned 8B (script 115)
8b_seed2_exp120	Qwen3-VL-8B-Instruct	second seed (120)
8b_qtext_exp120	Qwen3-VL-8B-Instruct	quality scores in the prompt (120)
8b_note_exp216	Qwen3-VL-8B-Instruct	full-reference evidence note in the prompt (216)
8b_note_s2_exp257, 8b_note_s3_exp257	Qwen3-VL-8B-Instruct	two more seeds of the note model (257)
8b_hybrid_note_exp259	Qwen3-VL-8B-Instruct	hybrid (reference + no-reference) note model, warm-started from 216 (259)
32b_cheap_exp117	Qwen3-VL-32B-Instruct	cheap LoRA 32B (117)

Final answer = per-option mean log-probability over option rotations, averaged over the 8B models (note models, qtext, seed2) and the 32B model, plus measurement-based window corrections. See the report for details.

Install

pip install torch==2.5.1 transformers==4.57 peft accelerate qwen-vl-utils
# ffmpeg/ffprobe are needed for frame extraction

Hardware: one 48 GB GPU (8B models in bf16); the 32B model needs about 2 x 48 GB or 4-bit loading.

Load an adapter

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
base = Qwen3VLForConditionalGeneration.from_pretrained("Qwen/Qwen3-VL-8B-Instruct", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "adapters/8b_note_exp216")   # path inside this repo
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")

Run on new questions

code/ holds the scripts used (numbered experiment scripts and test_pipeline/). Run from a copy of that folder with the adapters placed at outputs/<expNN>/fold0/adapter/ (for example adapters/8b_base_exp115 -> outputs/exp115_ft/fold0/adapter, as each script's docstring states):

CUDA_VISIBLE_DEVICES=0 python test_pipeline/run_test.py --name run1 --questions questions.json --videos-root <videos_dir>
python 216_finetune_reference_notes.py --predict-test      # note-model scores (needs notes from 215/258)
python 256_note_model_in_A.py --name out.zip --notes <note model output dirs>   # builds the final predictions.csv zip

Questions are JSON lists of {question_idx, video_idx, qa_category, question, candidates, answer} (answer is not read at inference). Output is predictions.csv with columns question_idx,predicted_option (1-4).

Exact reproduction of the final entry (B4), verified

repro_files/ holds the small intermediate files B4 needs (Test option log-probabilities of every model, the earlier Test entry test_v18_bundle.zip, Test questions, evidence notes, clean-source matches), with the same relative paths as in our working folder. To rebuild B4 without a GPU:

cp -r code/* .                    # all scripts into one folder (numbered scripts load each other)
cp -r repro_files/outputs .       # intermediate files
python 256_note_model_in_A.py --name B4_rebuilt.zip --notes outputs/exp216_refnote_back outputs/exp257_note_s2 outputs/exp257_note_s3 outputs/exp259_hybrid_warm

The result (outputs/submissions/B4_rebuilt.zip) is identical to our submitted test_v41_B4_hybrid.zip (also included, 300/300 answers equal; we ran this check on 10 Nov). To recompute the log-probabilities from the adapters, run the scripts named in each adapter's table row above (they need the Test videos, ffmpeg, and the frame extraction scripts in code/).