quote-and-retrieve-8b-grpo

Qwen/Qwen3-VL-8B-Instruct trained with a region-label-free GRPO recipe to quote better evidence when answering questions about long documents. This is the full merged model, ready to load directly; no adapter step and no base-model download are needed.

Paper: arXiv:2607.24651 · Code: github.com/Ryenhails/quote-and-retrieve · Eval set: Ryenhails/quote-and-retrieve-eval

What it does

The model does not output coordinates. It answers a document question and quotes its evidence verbatim as text. A layout parser and a multimodal retriever then turn each quote into a page region, outside the model. Training improves what the model chooses to quote, which is what the reward scores.

	base Qwen3-VL-8B (language interface)	this model
evidence recall @ IoU 0.5	39.9	51.3
strict attributed accuracy (SAA)	22.4	33.8
attribution hallucination (lower better)	41.2	28.4
same-page recall	63.2	73.8
citation precision	37.6	26.5
citations per response	2.1	4.6

Measured on the 719-question verified evaluation set with a Gemini-3.5-Flash judge. The gain is coverage-driven: the model quotes evidence it previously missed, and it also cites more regions, so precision falls. An independent judge that reads the retrieved crops confirms the improvement is real rather than reward gaming: under a stricter SAA variant that drops the recall disjunct and cannot be inflated by citing more, this model improves 19.1 → 27.4 (McNemar p = 6e-7).

Why "region-label-free"

The reward is built from the question, its gold answer, and rendered crops of whatever regions the retrieval step returned. A vision-language judge scores answer correctness, evidence relevance, and evidence coverage; the reward multiplies the answer term by the evidence terms, so evidence only counts once the answer is already correct. Nothing in that signal is a region label: the gold answer ships with any QA set, the retriever is unsupervised, and the judge sees only crops. This matters because region-level evidence labels are expensive to collect for long documents, which is what existing attribution systems train on.

Training

base model	Qwen/Qwen3-VL-8B-Instruct
method	GRPO, group-relative advantages, asymmetric clipping and dynamic sampling following DAPO
adaptation	LoRA r=64, alpha=64 on the language model, vision tower frozen, merged into the base weights for this release
data	1,584 questions built from LongDocURL, 83 held out; document-fingerprint checked for no overlap with the evaluation PDFs
rollout	8 samples per question at temperature 1.0
schedule	3 epochs, 144 steps; the final checkpoint is released, with no checkpoint selection
compute	about 85 GPU-hours on H200

Usage

The model expects the language-interface prompt. Its output is a JSON object with an answer and verbatim quotes; the quotes become regions only after the retrieval step in the code repository.

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

MODEL = "Ryenhails/quote-and-retrieve-8b-grpo"
processor = AutoProcessor.from_pretrained(MODEL)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL, dtype=torch.bfloat16, device_map="auto").eval()

SYSTEM = """# Document Analysis Assistant

Answer the question based on the provided PDF page images. For EVERY piece of evidence that supports your answer, QUOTE the exact text span from the document (verbatim, as it appears), so it can be located in the source.

## Rules
1. Each evidence quote must be a verbatim text span copied from the page (a sentence, a table row/cell, a caption, or a note). Do not paraphrase.
2. Quote the evidence at the element level: enough text to identify the region, not a single word and not a whole page.
3. If multiple regions support the answer, give multiple quotes.

## Output Format
Return a single JSON object exactly in this form:
{"answer": "<your full answer text>", "evidence": [{"page": p, "quote": "<verbatim text>"}, ...]}
where p is the 0-based page index in the order shown."""

messages = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM}]},
    {"role": "user", "content": [
        *[{"type": "image", "image": img} for img in page_images],   # all pages, in order
        {"type": "text", "text": "What specific alloy is used for the internal contact pins?"},
    ]},
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1536, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

{"answer": "The internal contact pins use a beryllium-copper alloy.",
 "evidence": [{"page": 1, "quote": "the internal contact pins are fabricated from a beryllium-copper alloy"}]}

Render pages at up to one megapixel each, in document order, matching the evaluation protocol. Greedy decoding with a 1536-token budget reproduces the reported numbers.

Turning quotes into regions

The quotes are only half the pipeline. To get boxes, run the retrieval step from the code repository:

git clone https://github.com/Ryenhails/quote-and-retrieve.git && cd quote-and-retrieve
python src/prebuild_block_emb.py      # parse pages into semantic blocks, embed each block crop
python src/match_wholedoc.py          # assign quotes to blocks one-to-one; writes pred_tuples
python src/score_citevqa.py outputs/matrix/<condition>/predictions.jsonl

End-to-end instructions, including how to combine this model with the evaluation set, are in docs/using_the_release.md.

Limitations

The model improves recall by quoting more, and citation precision drops from 37.6 to 26.5, because the reward has no precision term. It is a single seed on a single backbone. Attribution still depends on the external parser and retriever: the model itself never emits a region, so the achievable ceiling is the parser's (88.2 recall at IoU 0.5 on this evaluation set). Purely graphical evidence with no caption or nearby text cannot be quoted at all. A hallucinated quote is still assigned to some plausible-looking block rather than flagged, so deployments should expose the retrieval similarity as an abstention signal.

License

Released under Apache 2.0, matching the terms of the base model Qwen/Qwen3-VL-8B-Instruct whose weights this model contains.

Citation

@article{liu2026evidence,
  title         = {Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels},
  author        = {Liu, Zhuchenyang and Zhang, Yao and Xiao, Yu},
  journal       = {arXiv preprint arXiv:2607.24651},
  year          = {2026},
  eprint        = {2607.24651},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2607.24651}
}