Yamada114514/AliceAI-Foundation-80B-A3B-Base-MLX-4bit

AliceAI-Foundation-80B-A3B-Base — experimental MLX 4bit

An unofficial, experimental mixed-precision quantization and MLX port of Yandex's AliceAI-Foundation-80B-A3B-Base. Original model and architecture credit belongs to Yandex. This release provides quantized weights, a corrected independent MLX implementation, and limited local validation. It is a pretrained Base model, NOT instruction-tuned, NOT chat-ready and NOT production-ready. No fine-tuning or additional training was performed.

Source revision: b7984f62fd212535d2de9094bfeba23de58cd7c8. Test hardware: Apple M5 Max, 128 GB unified memory, macOS. Tests dated 2026-09-22.

Precision and contents

• MLX affine quantization, group size 64, predominantly 4bit; not a uniform 4bit model.
• Router weights are BF16; packed eligible dense/expert/embedding/output weights are Q4; normalization and other ineligible tensors retain their stored precision.
• Original router correction biases are retained. MTP tensors are excluded: MTP is not implemented / OFF.
• No dense-8 overlay is required. The published weights are the repaired r2 variant.
• Safetensors contain the same quantized weight bytes tested locally; runtime repair did not requantize them.
• model.py is required. This is not a claim of native support in upstream MLX-LM, Transformers, LM Studio, or an API server.
• Source tokenizer files are included, with legacy=false. No default chat template is installed.
• MLX-only config omits upstream Torch auto_map and sets mtp_num_hidden_layers=0 to avoid implying MTP support.

Run locally

Tested: Python 3.14, mlx==0.32.0, mlx-lm==0.31.3, transformers==5.16.1. Create an isolated environment. These commands use uv; do not upgrade an existing shared runtime.

# For a private repository, authenticate first using hf auth login.
uvx --from huggingface_hub==1.32.0 hf download Yamada114514/AliceAI-Foundation-80B-A3B-Base-MLX-4bit --local-dir ./aliceai-mlx-4bit
cd aliceai-mlx-4bit
# Review model.py and inference.py before executing this repository's custom code.
uv run --python 3.14 --with mlx==0.32.0 --with mlx-lm==0.31.3 --with transformers==5.16.1 python inference.py --model . --preset raw --max-tokens 96 --prompt 'The capital of France is'

inference.py calls mlx_lm.utils.load_model(..., strict=True), which executes the local model.py named by config.json. It uses a local LlamaTokenizer, one Hello warmup, greedy decoding, seed 0 and no speculative decoding. No source checkpoint download or parent experiment directory is needed. --json emits output and measurements. This CLI is bounded to 2,048 total input tokens and 1,024 output tokens; these limits are a safety bound for the example, not a long-context performance validation.

Optional Q4-selected calculation-oriented prompt

The runner defaults to --preset bilingual_checked. Supply an English question; the included examples request a brief Japanese answer. It does not translate your input.

uv run --python 3.14 --with mlx==0.32.0 --with mlx-lm==0.31.3 --with transformers==5.16.1 python inference.py --model . --prompt 'Available RAM is 96 GB. Weights require 58 GB, KV cache 17 GB and overhead 9 GB. State the total, remaining capacity and whether it fits, in Japanese.'

The exact prefix is in preset.txt. Live stopping checks after each token for a blank-line-separated next question or user/system marker and removes the marker. This is external stopping, not natural EOS; merely copying the prompt into a different application does not install this behavior. It can stop prematurely for quoted conversations or multiple-question outputs. Use --preset raw to disable it.

On a small, separate six-question Q4 confirmation set, the revised preset passed 5/6 versus 3/6 for its bilingual control. Three tasks were memory calculations close to the examples. A logical explanation failed. This is not an 83% general accuracy score. Final-preset coding and general-purpose Japanese quality were not established by that set. Q8 has not undergone this preset-selection study. Q4 candidate median decode was 48.07 tok/s, TTFT 0.937 s over that six-question set; all outputs stopped before 256 tokens, so a 1,024 ceiling itself provided no measured gain. Human translation time is excluded.

Packaging follow-up on Q4: the same final preset reproduced its earlier memory-fit answer exactly, correctly answered the previously troublesome 128 GB / 132 GB required / 4 GB shortfall task, and generated a count_positive function passing six restricted execution tests. These are additional spot checks, not an expanded independent accuracy benchmark. Q8's 96-token raw completion reproduced its original token sequence. Both release trees loaded without a source-directory dependency.

Short smoke measurements, not a comprehensive benchmark

Sequential Q4/Q8 runs; identical three short raw prompts (English, Russian, Japanese), greedy temperature 0, seed 0, one short warmup, 96-token output ceiling each, one measured run per prompt. Finite-logit checks at every step.

Metric	Q4	Q8
Local disk usage (GiB, rounded)	41.81	79.10
Load time (s, OS-cache dependent)	1.477	8.376
Decode range, three prompts (tok/s)	45.86–47.59	42.10–43.09
Japanese PP, 28 input tokens (tok/s)	243.17	220.43
Japanese TTFT (s)	0.115	0.127
Process peak RSS (GiB)	42.11	60.94
Peak MLX allocation (GiB)	42.10	79.40
System swap endpoint delta	0	0
Crash / non-finite logits	none observed	none observed
Natural EOS in these three runs	0/3	0/3

RSS and MLX allocation are different measures; Q8 does not fit in 61 GiB merely because RSS was 60.94 GiB. Load times are not cold-disk guarantees. PP is input tokens divided by first-token latency, not a long-input throughput benchmark. TTFT ends when the first token is evaluated; decode is subsequent 95 tokens/time. Swap was system-wide usage before/after, not per-process or transient peak swap. The small warmup does not eliminate compilation for all shapes. smoke_results.json contains the original measured outputs and sanitized evidence.

Runtime repair and validation

The initial independent port generated broken text due to runtime mismatches; this was not evidence that quantization itself was unusable. Corrections include:

1. Block-AttnRes ordinary RMSNorm must use weight, not zero-centered 1 + weight.
2. Four-layer block boundaries must preserve accumulated residuals, not revert to embeddings.
3. Sensitive normalization, KDA gates and router arithmetic were aligned with the reference.

Runtime SHA256: e6a9ef1c76aa4c7ae850e83d8521a093c1d9f7b03bdf158aaf001f7a8dcc7a10. Against the official Torch reference with identical randomized eight-layer test weights: CPU max logit difference approximately 2.38e-7; split-cache max difference 2.98e-7; Metal max difference 8.96e-4, mean relative difference 0.171%. An actual-weight first-layer check gave mean relative difference 0.0954%. This is not full 48-layer BF16 real-model equivalence validation.

Additional Q4 smoke: 17×23=391; a 120-second Amdahl task with 80% sped up 4× gave 48 seconds; an extracted generated Python function passed six independently executed cases. These are narrow sanity checks, not a reasoning/coding benchmark score.

Limitations and recommendation

• Base-model continuation, repetition, extra questions and failure to reach EOS remain.
• Instruction following is partial. Readable Japanese does not establish reliable Japanese capability; metadata lists only upstream-supported Russian and English. Some technical answers are wrong.
• No alignment or content-safety assurance. Independently validate any output before relying on it.
• Long context, agents/tool use, production serving, batching and cache serialization are unvalidated.
• The config inherits a 262,144 position limit; that is not a tested limit of this port.
• KDA prefill/preprocessing is a simple sequential implementation, not optimized.
• MTP is OFF/unimplemented. No speculative speedup is claimed.
• Q8 quality superiority over Q4 is not established. Q8 costs more memory and was slower in these short runs.
• Prefer Q4 for local experimentation; retain Q8 as a comparison artifact, not an assumed quality upgrade.
• Private uploads do not constitute a public release. Repository visibility is controlled separately.

License, attribution and modifications

The upstream model is Apache-2.0, Copyright 2026 YANDEX LLC. Original LICENSE and NOTICES are retained unchanged; the full license is also in APACHE-2.0.txt. Observe the export-control notice and applicable laws. Quantization, tensor layout conversion, excluded MTP tensors, MLX architecture translation, runtime corrections, tokenizer packaging, MLX config changes and this inference example are derivative modifications, dated 2026-09-22. Modified runtime files are marked. The added example/preset is provided under Apache-2.0. No Yandex endorsement or ownership of the underlying model is claimed. See conversion.json for provenance.

References: upstream pinned source, Apache-2.0.