Laya AI Model: How It Works, Run It Locally, and Evaluate It

The Laya AI model is built for a particular kind of work: making a bounded decision about an input, then returning a structured answer and probabilities. Give it a support ticket, an email, a conversation, or a JSON state. Ask which queue should receive it, how urgent it is, or whether a stated condition is true. Laya answers those questions without writing a paragraph that your application must parse.

That makes Laya interesting to teams that need local inference, open weights, and control over fine-tuning. It also creates a responsibility: you must measure whether its answers and confidence values are reliable for your own workflow. A valid output shape is not proof that a decision is correct.

This guide covers the model's interface and architecture, the available checkpoints, a runnable Python example, and a practical evaluation plan. It uses the upstream Laya repository and Convai Innovations model card as primary sources, checked September 26, 2026.

Table of contents

• Laya AI model at a glance
• How Laya turns state into typed decisions
• Choice, Score, and Noul explained
• Choose the right Laya checkpoint
• Run Laya locally with Python
• Design a decision task that can be evaluated
• Evaluate accuracy, calibration, and abstention
• Latency, operations, and total cost
• Try Laya AI online
• Laya AI model FAQ

Laya AI model at a glance

Laya is an open-weight, non-autoregressive decision-model family from Convai Innovations. Its input is a state plus one or more typed questions. Its output is an answer object with values and probability information for each question. The project's code and published weights use the Apache 2.0 license. The upstream package can run locally and provides a Router that selects among checkpoints for different inputs.

Think of a support queue. A generative model can write a helpful reply to the customer; a decision model can select a routing label and estimate escalation risk. Those are different contracts. Laya's typed questions can also support experiments in lead qualification, content review, and agent or model routing.

How Laya turns state into typed decisions

The English Laya checkpoint uses a bidirectional ModernBERT encoder with decision heads; the multilingual checkpoint uses mmBERT. In simplified terms, the runtime formats the state and each question, encodes that information, scores the permitted answers, and returns a distribution. There is no token-by-token generation of a response sentence.

This difference matters in an application. When a chat model produces "billing" in a JSON string, you still have to protect against invalid JSON, unexpected labels, extra text, and schema drift. Laya's interface makes the allowed answer space explicit before inference. You still need to validate the runtime response and apply your own business rules. The model's decision is an input to policy, not a substitute for policy.

For multiple questions, the upstream documentation describes a single batched forward pass. Do not assume that the state is encoded once and freely reused for every arbitrary question: implementation details and latency depend on the chosen runtime, question schema, and batch shape. Benchmark the exact version you deploy.

The crucial pipeline is:

state + typed questions → checkpoint → answer distributions → application policy

A good state contains the evidence needed for the decision, not merely a long dump of everything your service knows. For a duplicate-charge ticket, include the subject, message, account status, and any verified payment facts that matter. Keep secrets and irrelevant fields out. If your policy distinguishes a requested refund from an approved refund, encode that distinction in the question and enforce approval outside the model.

Choice, Score, and Noul explained

The three question types cover common application decisions. They are simple enough to reason about, but their quality depends on the criteria you write.

Choice: select from named options

Use choice when the answer must be one of a fixed set: billing, technical, sales, or other. Define each option with concrete criteria and decide what happens when none fit. Avoid overlapping labels such as urgent and important unless an annotator could consistently tell them apart. Choice returns a selected option and a distribution over the options, according to the upstream model card.

Example question: “Which team owns this issue based on the reported problem?” A ticket saying “charged twice” belongs under billing even if it also says the website is slow. Your routing policy can then use a secondary question to flag the outage separately. That is easier to audit than expecting one label to carry every meaning.

Score: rate on an ordered rubric

Use score for ordered levels such as routine, soon, and blocking. The order matters. Write descriptions that give annotators and the model a shared boundary. “Soon” should mean something operational, such as action within one business day, rather than a vague feeling. Laya returns a distribution over levels and an expected score.

The upstream project flags ordinal scoring as a weaker area in its published evaluation. Treat a score as a candidate signal until your own confusion matrix shows that the model distinguishes adjacent levels well enough.

Noul: estimate a proposition

Use noul for a yes/no proposition, represented as P(true). “Does the customer explicitly threaten to cancel?” is testable from the text. “Is this customer unhappy?” is broader and will be harder to label consistently. A value such as 0.83 is a model probability estimate for the stated proposition; it is not an 83% guarantee that the proposition is true in this individual case.

For all three types, separate prediction from action. A high churn-risk estimate might create a review task. It should not automatically issue a refund or change an account's permissions.

Choose the right Laya checkpoint

The Laya family publishes three checkpoints with different encoders and intended uses. Check the current model card before pinning a release, because context support and runtime options can change.

The upstream README says the multilingual checkpoint can accept up to 8,192 tokens when explicitly configured with max_len=8192. It also reports more variable accuracy on longer documents, especially beyond approximately 4,000 tokens in its small long-context evaluation. That is a capability to test, not a reason to assume full-document comprehension. The default limit still truncates longer inputs unless you override it.

The Router can choose the English or multilingual checkpoint based on language and script. Short Latin-script messages can be ambiguous, so a multilingual product should inspect routing metadata and consider Router(default="multilingual") or an explicit model choice. Pin the package and model revision for reproducible production results; recheck quality after an upgrade.

Run Laya locally with Python

The upstream quickstart requires Python 3.10 or newer. Install the package into a virtual environment, then run a small decision request. The first prediction downloads the required checkpoint from Hugging Face, so plan for that download and cache it in deployment environments.

python3 -m venv .venv
.venv/bin/python -m pip install laya

Create a script with one support-ticket state and three explicit questions:

from laya import Router

router = Router()

state = {
    "subject": "Duplicate charge",
    "body": "We were billed twice for March. Please refund the duplicate today or we will cancel.",
}

questions = {
    "department": {
        "type": "choice",
        "instructions": "Which team should handle this request?",
        "criteria": {
            "billing": "invoices, payments, refunds",
            "technical": "bugs, outages, errors",
            "other": "everything else",
        },
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgent is the request?",
        "criteria": ["routine", "soon", "blocking"],
    },
    "churn_risk": {
        "type": "noul",
        "instructions": "Does the customer explicitly threaten to cancel?",
    },
}

result = router.predict(state, questions)
print(result["answers"]["department"]["choice"])
print(result["answers"]["churn_risk"]["noul"])
print(result["routing"]["model"])

This follows the upstream Router.predict shape. The printed choice and probability are model outputs, so inspect the complete response and compare it with your expected labels. Do not hardcode example confidence values: they can change with the checkpoint, package version, device, and input wording.

For an HTTP service, the upstream repository also documents laya[serve] and a local JSON server. For Apple Silicon and Node.js, community runtimes exist, but verify their supported checkpoints, output parity, licensing, and maintenance separately. The upstream Python implementation is the clearest reference point for a first evaluation.

Design a decision task that can be evaluated

The fastest way to waste an evaluation is to ask an unlabelable question. Before comparing models, write down the input, approved answers, label definitions, and the action each answer can trigger. Let two humans label a small sample independently. Where they disagree, improve the rubric rather than assuming the model must resolve an unclear policy.

For support routing, a useful annotation sheet might record ticket ID, language, issue type, correct owner, urgency, cancellation threat, and reviewer notes. Split by customer or conversation where needed, so near-duplicate tickets do not leak across training and test sets. Keep a final test set untouched until you have selected prompts, thresholds, and any fine-tuning method.

Add cases that reflect real failure modes: short messages, misspellings, code-switching, irrelevant quoted text, contradictory evidence, many possible labels, and an other category. Also record the consequence of a mistake. A misrouted routine ticket costs time; an incorrect safety decision may require a stricter threshold and human review.

The upstream project itself warns that a large choice option set can degrade results because option text shares a fixed budget. If you have dozens of categories, try a coarse first choice followed by a smaller second choice, then measure whether the extra step actually improves accuracy and latency. Do not assume that a hierarchy is automatically better.

Evaluate accuracy, calibration, and abstention

Start with task metrics, not a single headline benchmark. For choice, report a confusion matrix and per-class precision and recall, especially for rare or costly labels. For score, inspect errors between adjacent levels and errors that jump multiple levels. For noul, use a precision-recall curve or another metric suitable for the class balance, then choose a threshold against the real cost of false positives and false negatives.

Calibration asks whether predicted probabilities match observed frequencies across many examples. If cases assigned roughly 0.8 probability are correct only half the time, the numbers are too confident for that task. Plot a reliability diagram and compute a calibration measure such as expected calibration error, but keep the binning method and sample size visible. The upstream documentation recommends fitting temperatures on held-out task data; any recalibration must use validation data rather than the final test set.

An abstention policy gives uncertain cases to a fallback. Instead of asking only “What is accuracy?”, ask “At 70% automatic coverage, what is the error rate on cases we automate?” Report both coverage and selective accuracy. A threshold can make the automated subset look better by moving difficult cases to humans; the cost and turnaround time of that review still count.

Measure each language, customer segment, and input length separately. A pooled average can hide a poor result on the very group that needs the workflow. Use shadow mode first: run the model beside the existing process, log its proposed output, and compare it with later verified outcomes without allowing it to take action. Promote one bounded action at a time.

Latency, operations, and total cost

Laya's upstream benchmarks report fast inference on specified GPU setups, but those numbers are not a universal service-level promise. Your users experience request serialization, queue time, model warm-up, inference, policy code, and the network hop to your application. Measure p50 and p95 under realistic concurrency, not just one warm call. Repeat the measurement for your actual checkpoint, context length, device, and number of questions.

Local weights remove a hosted model API dependency, yet they add deployment work: downloading and pinning checkpoints, keeping enough memory available, handling cold starts, monitoring errors, and rolling back versions when quality changes. CPU inference can be convenient for a small workflow, while sustained traffic may justify a GPU or a carefully measured batch server. Test throughput and failure recovery before estimating capacity.

Total cost includes hardware utilization, engineering time, annotation, fine-tuning, calibration, monitoring, and human review. A model that is cheap per inference can still be expensive if it routes too many tickets incorrectly. Conversely, a strong local model may be valuable when data residency or offline operation is a hard requirement. Use the same held-out cases and service conditions when comparing deployment options.

Try Laya AI online

To explore Laya without a local installation, open the Laya AI playground. Choose an example such as Support triage, Lead scoring, or Model routing, replace the input with your own sample, and adjust the choice options, scoring criteria, or yes/no question. Run the request to inspect the structured answers, probabilities, and request JSON.

The playground uses convaiinnovations/laya. Running a request requires signing in and consumes account credits; the returned result shows the actual credits used. The same page includes a Laya API integration guide for connecting the workflow to your application. Start with a task whose expected answer you know, then change one input or question at a time to see how the result responds.

Laya AI model FAQ

Is the Laya AI model open source?

The upstream repository and published weights carry an Apache 2.0 license. Review the license and the exact checkpoint you use before distribution. Open weights do not mean that every piece of training data or every training run is available for independent reproduction.

Can Laya run offline?

Yes, after you have installed the runtime and downloaded the checkpoint into the environment. A first load from the Hugging Face Hub needs network access unless you provide a local cached copy. Test a clean deployment in the same network conditions as production.

Does Laya generate text or JSON explanations?

Its core interface returns typed decision values and probabilities rather than free-form prose. If your product needs a customer-facing explanation, generate or write that separately and make sure it does not invent evidence that the decision input did not contain.

Is Laya's confidence the same as accuracy?

No. Confidence is an estimate attached to an individual prediction. Accuracy is measured over labeled examples. Calibration tests whether confidence levels correspond to observed correctness across groups of cases; both can shift when the data or model changes.

Which checkpoint should I use for another language?

Start with laya-multilingual and test the languages, scripts, and domain terms present in your traffic. The Router can select it automatically, but very short messages may need an explicit default or model override. Check truncation and quality for long inputs.

When should I fine-tune Laya?

Fine-tune after a baseline evaluation shows a repeatable gap and you have enough clean, representative labels. Keep a held-out validation set for threshold and temperature fitting and a separate final test set. Compare the specialized checkpoint with the untuned baseline under the same conditions.

Sources and update note

Architecture, installation, checkpoint choices, context limits, and known limitations were checked against the official Laya repository and official model card on September 26, 2026. Version-specific performance claims should be rechecked before publication or a production rollout.

Online examples, playground behavior, and access requirements were checked against the Laya AI page on October 3, 2026.