KDLAI

/

KDL-Frontier-Parser-nano

like 12

Follow

KoreaDeepLearning 16

Thanks for the model and for adding the Usage section. We followed the pipeline
(detect → crop → per-region call) and the recognition tasks work great, but the
Layout Detection: task never produces real regions on any of our documents —
it degenerates into uniform bands / marching identical boxes. We've tried to rule out
every runtime/flag variable we could and would love some guidance. Full repro below.

TL;DR

• \nText Recognition: and \nTable Recognition: on cropped regions → accurate (Table returns correct OTSL).
• \nLayout Detection: on a full page → degenerate every time: 88–93 identical boxes that either stripe the page full-width or march in a line, ignoring actual content. Happens under both vLLM and transformers, on 14+ documents including a dense in-distribution academic paper.
• Following the card's two "repeated-nonsense" fixes (enable_thinking=False, skip_special_tokens=False) does not change it — see "Why the two flags don't apply here".

Environment

(sm_120 note: vLLM's FlashInfer JIT sampler fails to build for sm_120 — "SM 12.x requires CUDA >= 12.9" → generic "FlashInfer requires GPUs with sm75 or higher". We disable it with VLLM_USE_FLASHINFER_SAMPLER=0. This is unrelated to the detection issue; the same degeneration reproduces on the pure-transformers path with no vLLM/FlashInfer involved.)

Serve command

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve KDLAI/KDL-Frontier-Parser-nano \
  --served-model-name kdl-frontier-parser-nano \
  --max-model-len 8192 --gpu-memory-utilization 0.6 --max-num-seqs 24 \
  --trust-remote-code --limit-mm-per-prompt '{"image":1}'

Exact request (OpenAI-compatible chat endpoint)

{
  "model": "kdl-frontier-parser-nano",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,<PAGE>"}},
    {"type": "text", "text": "\nLayout Detection:"}
  ]}],
  "temperature": 0,
  "max_tokens": 2048,
  "skip_special_tokens": false,
  "chat_template_kwargs": {"enable_thinking": false}
}

Fully rendered prompt the model receives (image pad collapsed):

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|image_pad|>…<|vision_end|>
Layout Detection:<|im_end|>
<|im_start|>assistant

What works — recognition (same endpoint, same flags)

\nText Recognition: on a cropped company-name region →

ACME ROBOTICS INC.

\nTable Recognition: on a cropped line-item table → correct OTSL:

<fcel>Description<fcel>Qty<fcel>Unit Price<fcel>Amount<nl>
<fcel>Vision LLM inference license<fcel>2<fcel>$1,200.00<fcel>$2,400.00<nl>
<fcel>On-prem GPU server (RTX 5090)<fcel>1<fcel>$3,500.00<fcel>$3,500.00<nl>
...

So the endpoint, chat template, image pipeline, flags, and decoding are all correct.

What fails — \nLayout Detection: (degenerate on every document)

The output token format is always well-formed
(<|box_start|>x1 y1 x2 y2<|box_end|><|ref_start|>LABEL<|ref_end|><|rotate_up|>,
coords normalized 0–999), but the content is degenerate. Three failure shapes:

(a) Full-width horizontal striping — e.g. a clean 1000×1300 invoice → 93 identical boxes:

<|box_start|>000 000 999 060<|box_end|><|ref_start|>header<|ref_end|><|rotate_up|>
<|box_start|>000 060 999 100<|box_end|><|ref_start|>header<|ref_end|><|rotate_up|>
<|box_start|>000 100 999 140<|box_end|><|ref_start|>header<|ref_end|><|rotate_up|>
... (marches by ~40, all label "header", loops after y=980 back to y=100)

(b) Marching specks — a dense 2-column academic paper (1224×1584) → 88 tiny boxes crawling across the top running-head, nothing for the title/abstract/columns:

<|box_start|>100 040 118 049<|box_end|><|ref_start|>page_number<|ref_end|><|rotate_up|>
<|box_start|>120 040 138 049<|box_end|><|ref_start|>page_number<|ref_end|><|rotate_up|>
... (marches in x, all "page_number")

(c) Zero parseable boxes — some photos return looped short text (e.g. 06\n06\n06…) instead of box tokens.

Across 14 documents (synthetic invoice, real Turkish invoices, bank cheques, retail receipts/fiş, a filled government form, and a dense academic paper), 0 produced a usable layout. Different images give different degenerate patterns (so the image IS being read — it's not a blank/misattached input), but never real regions.

Everything we tried (none fixed detection)

Why the two flags don't apply here

• skip_special_tokens=false is clearly in effect — the special layout tokens are present in the raw output.
• enable_thinking=false has nothing to disable: the repo's chat_template.jinja (both the live HF copy and our local one) contains no thinking block / no enable_thinking variable, and tokenizer_config.json (414 bytes) has no chat_template field at all. So there is no thinking mode to turn off in this repo.

Minimal reproduction

import base64, requests
URL = "http://localhost:8000/v1/chat/completions"
def call(png_path, prompt):
    b64 = base64.b64encode(open(png_path, "rb").read()).decode()
    body = {
        "model": "kdl-frontier-parser-nano",
        "messages": [{"role": "user", "content": [
            {"type": "image_url", "image_url": {"url": "data:image/png;base64," + b64}},
            {"type": "text", "text": prompt}]}],
        "temperature": 0, "max_tokens": 2048, "skip_special_tokens": False,
        "chat_template_kwargs": {"enable_thinking": False},
    }
    return requests.post(URL, json=body, timeout=300).json()["choices"][0]["message"]["content"]

print(call("any_page.png", "\nLayout Detection:"))     # -> degenerate striping / marching boxes
# For contrast, crop a single text/table region and:
# print(call("table_crop.png", "\nTable Recognition:")) # -> correct OTSL

Questions for the authors

1. Could you share one known-good Layout Detection: example — the exact input image plus the exact request (or fully rendered prompt string) and the expected box output? That would immediately tell us whether this is our invocation or a checkpoint issue.
2. Is there an image requirement for detection not in the card (min/max pixels, DPI, aspect ratio, RGB vs grayscale, page vs region)?
3. Is the shipped chat_template.jinja the one used for detection? (It injects system: You are a helpful assistant. and has no thinking block; tokenizer_config.json has no chat_template. Should a different template be present, and is a system prompt expected?)
4. What exact versions (transformers / vLLM) and decoding params was detection validated with? Is greedy correct for the Layout task, or does it need specific sampling?
5. Any known issue with the detection head at sm_120 / bf16 / this transformers version?

Happy to run any specific image + request you provide and report back. Thanks!