kaktus-v6

Document layout extraction with full text as JSON, using the 11 DocLayNet categories. Give the model a page image plus the task prompt below and it returns a JSON list with one entry per layout element:

[{"bbox": [124, 68, 383, 86], "category": "Page-header", "text": "Patent Application Publication"},
 {"bbox": [415, 245, 512, 272], "category": "Caption", "text": "## FIG. 7"},
 {"bbox": [174, 418, 758, 819], "category": "Picture"}]

• bbox is [x1, y1, x2, y2] in integers on a 0-1000 grid, origin top-left.
• category is one of: Caption, Footnote, Formula, List-item, Page-footer, Page-header, Picture, Section-header, Table, Text, Title.
• text is Markdown for text elements, HTML for tables, LaTeX with $...$ or $$...$$ delimiters for formulas, and fenced code blocks for code. Picture entries have no text field. Elements come in reading order.

The text prompt is required. The output is a bare JSON list, although the prompt asks for a single JSON object.

This is a full fine-tune of lightonai/LightOnOCR-2-1B-bbox-base (1B parameters), whose native 0-1000 coordinate grounding matches the label format.

Usage

Requires transformers from source (pip install git+https://github.com/huggingface/transformers). The checkpoint was saved with transformers 5.16.0.dev0.

import torch
from PIL import Image
from transformers import LightOnOcrForConditionalGeneration, LightOnOcrProcessor

PROMPT = """Please output the layout information from the PDF image, including each layout element's bbox, its category, and the corresponding text content within the bbox.

1. Bbox format: [x1, y1, x2, y2]

2. Layout Categories: The possible categories are ['Caption', 'Footnote', 'Formula', 'List-item', 'Page-footer', 'Page-header', 'Picture', 'Section-header', 'Table', 'Text', 'Title'].

3. Text Extraction & Formatting Rules:
    - Picture: For the 'Picture' category, the text field should be omitted.
    - Formula: Format its text as raw LaTeX with delimiters (`$...$` or `$$...$$`). Do not replace formulas with prose or placeholder tokens.
    - Table: Format its text as valid HTML. Preserve visible table titles/captions as part of the table block when they are clearly attached to the grid.
    - Code or terminal transcript regions: use fenced Markdown code blocks with a single language label when inferable (for example ```python, ```json, ```cpp, ```fortran). Put the opening fence, body, and closing fence on separate lines, and do not add HTML syntax-highlighting tags inside code.
    - All Others (Text, Title, etc.): Format their text as Markdown. Preserve existing headings and semantic inline styling unless the visual page clearly contradicts it. Apply Markdown `**...**` for bold, Markdown `*...*` for italic, and `<s>`, `<sup>`, and `<sub>` when clearly visible; treat underline/mark as lower-priority and avoid churn when uncertain. Use `<s>...</s>` only when a horizontal line crosses through letters/numbers at the x-height/midline.
    - Semantic styling is content fidelity when visually significant: visible headings, bold, strikethrough, superscript/subscript, code blocks, or formulas emitted as plain text are incomplete, but uncertain styling should be left unchanged.

4. Constraints:
    - The output text must be the original text from the image, with no translation.
    - Do not add commentary or explanations that are not visible in the source.
    - Preserve standalone identifiers/codes exactly, even when small (for example product IDs, reference codes, docket numbers, or PH134503).
    - All layout elements must be sorted according to human reading order.

5. Final Output: The entire output must be a single JSON object."""

model = LightOnOcrForConditionalGeneration.from_pretrained(
    "4thel00z/kaktus-v6", torch_dtype=torch.bfloat16
).to("cuda")
processor = LightOnOcrProcessor.from_pretrained("4thel00z/kaktus-v6")

prompt = processor.apply_chat_template(
    [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": PROMPT}]}],
    tokenize=False,
    add_generation_prompt=True,
)
image = Image.open("page.png").convert("RGB")
inputs = processor(text=[prompt], images=[image], return_tensors="pt", size={"longest_edge": 1232}).to("cuda")
inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=4000, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Also servable with vLLM:

vllm serve 4thel00z/kaktus-v6 --limit-mm-per-prompt '{"image": 1}'

Render pages at 1200-1550px on the longest edge; training and evaluation used 1232px. Dense pages can use the full 4000-token budget.

Tasks

Three prompts were trained. The first is the main task and the one evaluated below.

1. Layout with text: the prompt shown in Usage. Output as shown at the top.

2. Layout only: the prompt below. The trained answers use the same bbox ([x1, y1, x2, y2]) and category keys as the main task and no text field, so expect that format rather than the key names the prompt mentions.

   Please output the layout information from the PDF image, including each layout element's bbox and its category.
   
   1. Bbox format: "box_2d": [y1, x1, y2, x2], integer coordinates normalized to a 1000x1000 grid. Use the key "label" for the category.
   
   2. Layout Categories: The possible categories are ['Caption', 'Footnote', 'Formula', 'List-item', 'Page-footer', 'Page-header', 'Picture', 'Section-header', 'Table', 'Text', 'Title'].
   
   3. Output only "box_2d" and "label" for each element. Do not emit a text field.
   

3. Table to HTML: prompt Convert the image's table data into the HTML structure. with a cropped table image. Output is an HTML <table>.

Training

8x A100 80GB.

Evaluation

Layout detection is scored as F1 at IoU 0.5 with greedy matching; greedy decoding, 4000 new tokens, the main-task prompt. Predicted categories are mapped onto each benchmark's label set before scoring.

OmniDocBench, English subset (755 pages): F1 0.293 (precision 0.342, recall 0.256), mean IoU of matched pairs 0.701, class accuracy on matches 0.935, valid JSON on 83.6% of pages. Per-class recall: table 0.506, figure 0.434, page header 0.394, text 0.302, title 0.203. The benchmark's code and page_number classes have no DocLayNet counterpart and count as misses.

Internal held-out set (1,652 pages labeled with a 17-type schema): F1 0.325 (precision 0.270, recall 0.408), mean matched IoU 0.739, valid JSON 94.6%. Class accuracy on matches is 0.782; 6 of the 17 types have no DocLayNet equivalent.

The layout-only and table-to-HTML tasks have not been evaluated separately.