Sqwish/Kettei-Core

Kettei Core

Kettei Core scores supplied alternatives against a question and its context. Its reference interface returns a selected option and normalized scores for all options. Example applications include support routing, explicit rule checks and tool selection.

Property	Value
Parameters	5,058,255,360
Approximate BF16 weight size	10.12 GB
Architecture	K2HorizonForCausalLM
Reference request limit	4,096 input tokens by default; overlong inputs rejected
Output	Selected option, index, option scores, logits and input-token count

Weight bytes are not a runtime-memory requirement. No hardware latency or task-quality claim is made by these figures.

Use the reference interface

Download the reference wheel and source/evaluation archive.

After downloading the wheel, open a terminal in its directory and install it:

python -m pip install './kettei-0.1.0rc5-py3-none-any.whl[torch]'

Install the accompanying Kettei reference package. Load a complete local checkpoint directory or this model repository by the release tag, recording the resolved commit SHA for reproducibility. The reference package supplies the decision renderer; a generic chat template is not a substitute. The example downloads the pinned checkpoint on first use and reuses the local Hugging Face cache afterward. If Hugging Face requests authentication, run hf auth login in your terminal before loading; do not put access tokens in code.

from kettei import Kettei

model = Kettei.from_pretrained(
    "Sqwish/Kettei-Core",
    revision="v1.0",
    variant="core",
    device="cuda:0",
    dtype="bfloat16",
    trust_remote_code=True,
)
result = model.predict(
    context="The customer was charged twice for one order and wants a refund.",
    question="Which support queue should receive this request?",
    options=["Billing", "Technical support", "Shipping"],
)
print(result.to_dict())

Actual output for the example above on the tested CUDA/BF16 configuration (floating-point details may vary by environment):

{
  "choice": "Billing",
  "index": 0,
  "options": [
    "Billing",
    "Technical support",
    "Shipping"
  ],
  "scores": [
    0.9732547402381897,
    0.014847113750874996,
    0.011898222379386425
  ],
  "logits": [
    28.1040096282959,
    23.92116928100586,
    23.699752807617188
  ],
  "input_tokens": 69
}

The custom architecture requires model code execution. Review the supplied code before enabling trust_remote_code=True. For reproducibility, use the resolved immutable checkpoint commit or a verified local snapshot. The CUDA/BF16 reference path was tested on NVIDIA RTX 6000 Ada with PyTorch 2.8.0+cu129 and Transformers 5.17.0.

Score interpretation and limitations

Scores are normalized preferences among the offered choices, not calibrated correctness or action-success probabilities. Changing the choices can change every score. The correct answer may be missing from the list. This interface does not generate explanations.

Evaluate candidate ordering, wording, ambiguous inputs and out-of-domain examples before using the model in a workflow. Application-specific abstention or review rules require evaluation on held-out data. General language coverage, universal calibration and performance on untested hardware are not established by the released diagnostic.

Latest measured results

On the original fixed DI-S12k sample (12,000 requests / 88,789 questions), Dot and Core scored 27.98 and 46.28 under the historical 0.2.1 skill-index formula. These are index points, not percentage accuracies or official current leaderboard scores. Task-macro NLL / Brier / ECE10 were 1.1464 / 0.5513 / 0.1281 for Dot and 0.9476 / 0.4544 / 0.1023 for Core, over 41 eligible tasks. The accompanying evaluation record includes the complete scope and the BANKING77 overlap finding.

The following are separate, completed evaluations under fixed native single-read recipes; they are not a combined score or a claim of universal superiority.

Model	Correct / 231	Accuracy	NLL (nats)	Brier	ECE10
Kettei Dot	156	67.53%	0.7524	0.4118	0.0681
Kettei Core	198	85.71%	0.3711	0.2035	0.0496
D3 Mini	189	81.82%	0.4162	0.2402	0.0678

Jev231 is the frozen downloadable public corpus, not the hosted/private leaderboard. All models completed every case on M4 24 GiB; canonical models used BF16 bodies and FP32 readout. Core–Mini accuracy difference is +3.90 points, with an exploratory paired 95% interval of -1.30 to +9.09. NLL and Brier difference intervals also include zero.

Local AA component and primary denominator	Dot	Core	D3 Mini
Omni accuracy, 600	11.50%	24.17%	34.50%
HLE accuracy, 513	18.13%	15.59%	11.89%
When2Call accuracy, 3,423	57.64%	62.02%	77.30%
WildGuard macro F1, 2,396 fields / 801 requests	84.96%	91.74%	95.21%

The local AA reconstruction completed 5,588 requests / 7,227 questions per model with zero errors on A100-SXM4-40GB. It differs from the official AA benchmark; no AA index is claimed. Mini leads Omni, When2Call and WildGuard point estimates; Dot leads HLE. The primary component denominators differ and are not pooled. Full-input secondary scopes, probability metrics, environment and paired uncertainty are documented in the accompanying reference archive's source/evaluation archive (docs/evaluation.md and docs/current-benchmark-aggregates.json). NLL/Brier measure probability quality; descriptive ECE does not establish application calibration.

Training provenance and historical diagnostics

A complete public training-data and training-procedure report is not included. The older 600-case synthetic evaluations and 767-example public-data diagnostic remain in the accompanying source/evaluation archive, under docs/evaluation.md, including unfavorable results, hardware differences and fixed-recipe limitations. They are not replaced by the newer results.

The training corpus associated with both checkpoints includes BANKING77-, SNLI-, and SST-2-derived material. A bounded audit found normalized exact whole-utterance matches for 252 of the 256 BANKING77 input messages, including the same 252 in upstream-test-labelled corpus rows. Normalization used Unicode NFKC, casefolding, and collapsed whitespace after removing the corpus message wrapper. These are input-message matches, not proof of identical questions, options, or targets; exact sampler consumption by each checkpoint has not been reconstructed. Full row-level and near-duplicate decontamination of SNLI and SST-2 has not been established, and foundation-model pretraining overlap is unknown. Treat this suite as a reproducible diagnostic, not a verified held-out generalization benchmark or evidence of an advantage on unseen data.

A separate audit of the original DI-S12k BANKING77 sample found normalized exact whole-message presence for 221 of 234 inputs in the same bounded checkpoint-associated corpora; all 221 also match upstream-test-labelled rows. Raw exact matching after wrapper removal found 214/234. This is distinct from the older 252/256 eight-intent diagnostic above. The comparison removes the known wrappers, then applies Unicode NFKC, casefolding and collapsed whitespace. Corpus presence does not prove sampler consumption by Dot or Core, nor identical full questions, alternatives or targets. The original benchmark membership and scoring remain unchanged. Other S12k tasks and D3 training overlap were not established by this audit; no clean-held-out claim follows.

License

Kettei is released as open weights and source available under the custom Kettei model license, not as OSI-approved open source. Free commercial use is available through US$1 million in annual revenue; organizations above that threshold need a commercial agreement. Read LICENSE-MODEL.md for the controlling definitions and terms, including attribution, redistribution and eligibility.

Apple silicon

The accompanying archive includes an experimental native MLX package for Apple silicon, with explicit local loading and a standard single-request path. It was tested on M4 24 GiB and M4 Pro 48 GiB. See mac_runtime/README.md for installation, measured task results and limits. Quantization and shared-context experiments are opt-in; none is the default.