NikhilKanta/vision_laya

Vision-Laya (Vision-Jev) 👁️⚡

Vision-Laya is a multimodal, non-autoregressive System 1 decision model built on top of convaiinnovations/laya.

Unlike generative Vision-Language Models (e.g., LLaVA, Qwen2-VL) that autoregressively generate tokens one by one over several seconds, Vision-Laya evaluates an image and typed questions in a single forward pass (~35–50 ms). It outputs mathematically calibrated probabilities, categorical choices, ordinal scores, and binary signals with zero hallucination risk.

---

Model Checkpoints

File	Size	Architecture & Capabilities
vision_laya_smollm2_f16.gguf	1.29 GB	Unified Multimodal Model: SigLIP ViT + Laya Decision Head + SmolLM2-135M Language Generation for fluid conversational descriptions and natural language visual QA.
vision_laya_f16.gguf	984 MB	Pure Decision Model: SigLIP ViT + Laya ModernBERT for ultra-fast non-autoregressive System 1 decisions (choice, score, noul).

• Vision Backbone: google/siglip-base-patch16-224
• Decision Backbone: convaiinnovations/laya (ModernBERT-large)
• Language Generation Head: HuggingFaceTB/SmolLM2-135M-Instruct
• Format: GGUF v3 (F16 precision)
• License: Apache 2.0

---

Why Vision-Laya?

Feature	Generative VLMs (LLaVA, Qwen2-VL)	Vision-Laya
Latency	500 ms – 3,000 ms (multi-token decode)	~35 ms (single forward pass)
Output Type	Freeform text string	Typed JSON (choice, score, noul)
Hallucination Risk	High	Zero (never generates text)
Confidence & Probabilities	Uncalibrated token logits	Calibrated probabilities (RLCD)
Hardware	High-end GPU / VRAM	Runs on laptop CPU or lightweight GPU

---

Visual Token Compression for Low Latency

Standard ViTs produce 196 or 576 patch tokens per image. Vision-Laya incorporates an adaptive Visual Compressor that pools visual features into 16 or 32 semantic tokens. This reduces cross-modal self-attention computation by over 10×, making laptop CPU inference blazingly fast.

---

Quickstart

from PIL import Image
from vision_laya import VisionLayaAgent

# 1. Load model (runs smoothly on CPU or GPU)
agent = VisionLayaAgent.from_pretrained(
    model_path_or_id="NikhilKanta/vision-laya",
    vision_model_id="google/siglip-base-patch16-224",
    device="cpu",
)

# 2. Input image & state context
image = Image.open("sample_document.png")
state = {"channel": "email_intake", "vendor": "Acme Corp"}

# 3. Define typed questions with instructions and criteria
questions = {
    "document_type": {
        "type": "choice",
        "instructions": "What type of document is shown in this image?",
        "criteria": {
            "invoice": "invoice with itemized line items and amount due",
            "receipt": "store checkout register receipt",
            "id_card": "passport, driver license, or employee badge",
            "other": "diagram, landscape, or unrelated document",
        },
    },
    "urgent_notice": {
        "type": "noul",
        "instructions": "Does the document display an urgent past-due or final notice warning?",
    },
    "quality_score": {
        "type": "score",
        "instructions": "Rate the visual legibility and clarity of the document.",
        "criteria": ["unreadable / blurred", "acceptable", "crisp and high clarity"],
    },
}

# 4. Predict in ONE single forward pass (~35 ms)
result = agent.predict(image=image, state=state, questions=questions)

print(result["answers"]["document_type"]["choice"])       # -> 'invoice'
print(result["answers"]["document_type"]["confidence"])   # -> 0.942
print(result["answers"]["urgent_notice"]["noul"])         # -> 0.885 (probability true)
print(result["answers"]["quality_score"]["score"])        # -> 1.95 / 2.0

---

GGUF Inspection

To inspect the vision_laya_f16.gguf file using Python:

pip install gguf
python -c "import gguf; reader = gguf.GGUFReader('vision_laya_f16.gguf'); print('Tensors:', len(reader.tensors))"

---

Citation & Acknowledgments

• Built on top of Laya by Convai Innovations.
• Vision backbone powered by Google SigLIP.
• Text backbone based on ModernBERT by Answer.AI.