Emerald7664/laya-multilingual-onnx

Laya Multilingual — CPU-Optimized ONNX Checkpoints

This repository provides high-speed, CPU-optimized ONNX checkpoints for convaiinnovations/laya-multilingual (322M parameters, mmBERT architecture).

These weights are engineered specifically for real-time prompt classification, semantic decision routing, and System One fast-path inference with 0.0 ms network latency on commodity CPUs.

---

Why CPU-Optimized Checkpoints?

The community WebGPU export (mizchi/laya-multilingual-onnx) is formatted in FP16. On standard CPUs without native half-precision vector execution units (such as x86_64 or ARM Neoverse N1/Cortex-A76), ONNX Runtime emulates FP16 in software, resulting in ~3,900 ms latency per sequence.

This repository provides two calibrated alternatives:

Checkpoint File	Precision	RAM / Disk	CPU Latency	Optimization & Target Hardware
model.onnx (Default) / model-int8.onnx	QInt8	325 MB	250 – 300 ms	Dynamic per-channel quantization. Accelerated via ARMv8.2-A asimddp instructions (sdot/udot) and x86 AVX-512 VNNI. Sensitive layers (scorer, act_head, type_emb) remain in full precision to preserve 100% classification accuracy.
model-fp32.onnx	Float32	1.29 GB	550 – 650 ms	Full precision FP32 weights. 100% bit-exact parity with PyTorch. Optimized for ARM NEON and x86 AVX2.
Original WebGPU export	Float16	617 MB	~3,900 ms	Emulated via software on CPUs lacking native FP16 instructions.

---

Files in this Repository

• model.onnx: The selective INT8 model (325 MB). Default for instant, low-latency CPU routing.
• model-int8.onnx: Explicit alias of the INT8 model.
• model-fp32.onnx: The full-precision FP32 model (1.29 GB).
• tokenizer.json & tokenizer/: Fast tokenizer files for Hugging Face tokenizers.
• rl_agent_config.json: Calibrated temperature scaling and category thresholds.

---

Quickstart with OpenProxy

OpenProxy automatically detects these models for native, in-process combo decision routing.

1. Download into the default OpenProxy directory

mkdir -p ~/.openproxy/models/laya
huggingface-cli download soyelmismo/laya-multilingual-onnx \
  --local-dir ~/.openproxy/models/laya \
  --include "model.onnx" "tokenizer.json" "rl_agent_config.json"

Or via direct HTTP curl:

mkdir -p ~/.openproxy/models/laya
HF_BASE="https://huggingface.co/soyelmismo/laya-multilingual-onnx/resolve/main"

curl -L -o ~/.openproxy/models/laya/model.onnx "$HF_BASE/model.onnx"
curl -L -o ~/.openproxy/models/laya/tokenizer.json "$HF_BASE/tokenizer.json"
curl -L -o ~/.openproxy/models/laya/rl_agent_config.json "$HF_BASE/rl_agent_config.json"

2. Run in Docker

Mount the models folder into the container:

services:
  openproxy:
    image: ghcr.io/soyelmismo/openproxy:latest
    ports:
      - "8787:8787"
    volumes:
      - ./config.toml:/etc/openproxy/config.toml:ro
      - ~/.openproxy/models/laya:/var/lib/openproxy/models/laya:ro

OpenProxy scans /var/lib/openproxy/models/laya and initializes native in-process inference without any Python dependencies or daemons.

---

License & Attribution

• Base Model: convaiinnovations/laya-multilingual
• Architecture: mmBERT (322M parameters)
• License: Apache 2.0