Sabaki-Overfit

Sabaki-Overfit is an experimental checkpoint based on our preview pretrained model, released to explore how does Sabaki's highly sparse design respond to posttraining deliberately focused on memorization?

This is a test release, not a general-purpose assistant or a replacement for Sabaki Preview. The objective was reproduction of training responses, rather than generalization to unfamiliar questions.

Purpose of this experiment

Our past attempts to posttrain Sabaki were not successful due to the highly sparse design causing the model to due poorly with sft training. Our fix we discovered was to prioritize memorization also called overfitting. Our hypnosis was that because the model has the ability to very small details the overfitting would be distributed over the experts and improve generation over a more standard approach

Release files

File	Description
Sabaki-Overfit.gguf	F16 GGUF export with the sabaki architecture identifier
Sabaki-Overfit.pt	PyTorch training checkpoint
Sabaki-Overfit-tokenizer.json	Tokenizer for the native checkpoint; also embedded in the GGUF

Running the model

Sabaki uses a custom architecture. The companion runtime is Sabaki Preview llama.cpp; use a version that supports the sabaki architecture identifier. See that repository for build and launch instructions, and select Sabaki-Overfit.gguf as the model.

Process

This model was trained using a similar process to the BananaMind-2-Pro-Thinking model but with some other differences than just more training.

Example response

The following response are an illustrative example, not a verified held-out evaluation. These questions did not appear inside of the training data.

Question: @user3616791 name a state in the USA.

Response:

<think>
We need answer user's request: "@user3616791 name a state in the USA". Need likely produce a simple US state name. Need final concise. Could be "New York" or "US". User asks simple. Provide simple name. Ensure maybe no extra.
</think>

New York

The response is reproduced without correction.

Failure example: factual explanation

This response is incorrect but shows the thinking process of the model on a more challenging question

Question: Explain why the sky appears blue.

Response:

<think>
Here's a thinking process:

    Analyze User Input:

    Question: "Explain why the sky appears blue."
    Key Concept: Blue light (specifically visible blue at the top)
    Goal: Provide a clear, concise, explanation of why the sky is blue, likely with a brief explanation of why it's not blue (as requested) and possibly mention its context.

    Identify Core Information Needed:

    Target light color: The Sun produces blue light.
    Target blue light: Red, Blue, Green, and Blue.
    Why it's not blue: Because it's a reference point-soil coloration effect; when a star looks blue through a window, it's not because the Sun is blue (in the sense of being visible from Earth) or because at any given moment, it's actually blue to the eye.
    Context/Usage: This is a common knowledge question. The Sun appears blue because we see objects that appear blue like stars, planets, and other celestial bodies.

    Structure the Explanation:

    Start with a direct, clear statement of the color.
    Explain the mechanism (sunlight → refraction/refraction of visible spectrum).
    Clarify that the sky is blue because objects in the sky are visible to the human eye, not blue into the eye.
    Keep it concise as instructed.

    Draft the Response (Mental Refinement):
    The sky appears blue because when you look at a star, it reflects mostly unvisible light that travels through the sky. When you see a star, it reflects mostly not back to the sun, just as the sun stays visible. This is why we call the sky blue—it’s because from a different angle, our eyes are focused on the Sun, which is actually blue to the eye.

    Check Against Constraints:

    Explains why sky appears blue? Yes.
    Concise? Yes.
    Avoids repetition? Yes.
    Accurate? Yes, aligns with scientific understanding.
    Language: Clear, direct.

All good. I'll output this.✅
</think>

The sky appears blue because when you examine a star, it reflects mostly unvisible light that travels through the sky. When you see a star, it reflects mostly not back to the sun, just as the sun stays visible. This is why we call the sky blue "blue."

Why is blue blue? When you view a star, the light appears to travel through the sky. From a different angle, our eyes focus on the Sun, which is actually blue to the eye. This is why the Sun is blue to the eye, and all the other light is mostly unvisible back to us.

Limitations

The pretrain that was used for this model is not the same one in the Sabaki Preview, this pretrain had about 16B pretrain tokens while the current preview only had 8B.

This checkpoint may reproduce familiar responses while failing on small changes to the question. It can also produce incorrect facts, inconsistent reasoning, unrelated text, or incomplete answers.

This highly overfit model outputs on average more on topic and correct reasoning compared to our non released post trains of Sabaki but still fails about the same on factual information.

Use it for local experimentation and research into sparse model behavior. It is not intended for production use, autonomous decisions, or high stakes advice.

Distillation

No close source or models being run via a 3rd party was used to generate or change the thinking or answer data. Models used for thinking trace distillation were Qwen3.6-35B-A3B and Qwen3.8-27B both quantized to AWQ-in4 being ran on our own hardware.

License

The model weights retain the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license used by Sabaki Preview. Software and third-party materials retain their respective licenses.

Attribution

If you use this checkpoint in research, credit Local-Axiom-AI and identify the model as Sabaki-Overfit. Distinguish results for this experimental checkpoint from those reported for Sabaki Preview.