Inferno-Exp-1.7

Inferno-Exp-1.7 is AlgoDriveAI's 393.6M-parameter, math-first decoder-only base language model. This release publishes the selected 32_indx.pt snapshot at continuation step 275,000, with 4,096-token context and 85.70% ArithMark 3.0 normalized accuracy. It continues the Deep400M training line represented by Inferno-Exp-1.6. It is a base text-completion model; it has no chat template or instruction tuning.

Property	Value
Unique parameters	393,610,240
Layers × width	32 × 1024
Attention	GQA: 16 query / 4 KV heads, head dimension 64
Position encoding	Full-head, half-split RoPE; base 10000
Feed-forward	Canonical same-token SwiGLU; inner width 2816
Context	4,096 tokens
Vocabulary	32,064; Mistral tokenizer with training additions and padding
Embeddings	Input/output weights tied
Released weights	Float32 Safetensors
Source checkpoint	32_indx.pt, step 275,000, saved epoch index 1

Parameter accounting: 32,833,536 tied token-embedding parameters + 360,775,680 parameters across 32 transformer blocks + 1,024 final normalization parameters = 393,610,240. The LM head is not counted twice.

Compared with Inferno-Exp-1.6

Benchmark	Inferno-Exp-1.6	Inferno-Exp-1.7	Change (points)
ArithMark 3.0 (acc_norm)	83.10%	85.70%	+2.60
HellaSwag (acc_norm)	29.80%	33.68%	+3.88
ARC-Easy (acc_norm)	47.01%	51.60%	+4.59
ARC-Challenge (acc_norm)	30.12%	34.04%	+3.92
PIQA (acc_norm)	60.45%	64.47%	+4.02

The ArithMark result was verified during this export. The four language scores are the full-set training evaluations recorded at the same step, 275,000. They are not a new post-export harness run. The previous release reports language evaluation batch size 16; the training evaluations here use 8.

Using these rounded scores and the training script's Open SLM Intelligence Index formula gives 32.02. This is a locally calculated summary, not an official leaderboard submission or rank.

Usage

# pip install "transformers==4.57.6" torch sentencepiece protobuf safetensors
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "AlgoDriveAI/Inferno-Exp-1.7"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True,
                                            torch_dtype=torch.float32)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()
inputs = tok("Sam had 47 marbles. His sister gave him 28 more. Now Sam has",
             return_tensors="pt", add_special_tokens=False).to(device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=80, do_sample=False,
                            use_cache=False, pad_token_id=tok.pad_token_id)
print(tok.decode(output[0], skip_special_tokens=True))

The custom Transformers implementation is reused from the pinned 1.6 release, with this checkpoint's 4,096-token configuration and new weights. The export checks strict tensor loading, compares its logits with the supplied trainer, reloads the saved Safetensors, and scores the exported model on ArithMark. Generation re-reads the context each step; no KV cache is implemented.

Try it: interactive demo (Gradio)

Run a small web app with a prompt box, sampling sliders, live streaming, a Stop button, and token/speed readouts:

1. Open a new notebook at Google Colab and choose a GPU (Runtime → Change runtime type → T4 GPU). CPU also works, but generation is slower.
2. Expand the script below, paste it into one code cell, and run it.
3. Open the *.gradio.live link printed by the cell.

The script loads the published Inferno-Exp-1.7 checkpoint and reads its 4,096-token context from the model configuration. The public model needs no Hugging Face token. If the cell asks for a restart after changing Transformers, use Runtime → Restart session and run that cell again.

Show the Gradio demo script (one Colab cell)

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Inferno-Exp-1.7 — simple Gradio inference (loads like any Hub user, no pushing)
==============================================================================
Loads Inferno-Exp-1.7 with plain Transformers (AutoModelForCausalLM +
trust_remote_code=True) and serves a small Gradio app: open prompt box,
sampling controls, streaming output with a Stop button, token/speed readout.

Works with either source:
  • the Hub repo:          MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.7"
  • the local export folder the push script builds (test before uploading):
                           MODEL_SOURCE = "/content/Inferno-Exp-1.7-release"

The public 1.7 model needs no token. For a private alternative, use an existing
Hub login, set the HF_TOKEN environment variable, or set HF_TOKEN below.

It is a base model: it continues text, it does not chat. Write the start of the text
you want (e.g. a word problem followed by "Answer:").
Hardcoded constants, no argparse. Paste into one Colab cell and run; open the share link.
"""
# =============================================================================
# SETTINGS — edit these
# =============================================================================
MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.7"   # Hub repo id (or a local folder with the same files)
MODEL_REVISION = "a24c67451dc7df3b3069e9ce54724ca815d96fda"  # verified 1.7 weights
HF_TOKEN = ""                                  # optional; otherwise use env var / existing Hub login
MODEL_DISPLAY_NAME = "Inferno-Exp-1.7"

# Default slider values (all adjustable in the UI)
DEF_MAX_NEW_TOKENS = 256
DEF_TEMPERATURE = 0.8
DEF_TOP_K = 40
DEF_TOP_P = 0.95
DEF_REPETITION_PENALTY = 1.1
DEF_SEED = -1             # -1 = different every run
DTYPE = "auto"            # "auto" (bf16 on newer GPUs, fp32 on CPU) / "float32" / "bfloat16" / "float16"

GRADIO_SHARE = True       # public *.gradio.live link (needed in Colab)
GRADIO_PORT = 7860
INSTALL_DEPENDENCIES = True

# =============================================================================
# RUN (no edits needed below)
# =============================================================================
import os, sys, time, threading, subprocess, queue
os.environ.setdefault("USE_TF", "0")
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")

try:
    import google.colab  # noqa: F401
    IN_COLAB = True
except ImportError:
    IN_COLAB = False
if IN_COLAB and INSTALL_DEPENDENCIES:
    import importlib.metadata as _md

    def _version(name):
        try:
            return _md.version(name)
        except _md.PackageNotFoundError:
            return None

    _before = _version("transformers")
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--quiet",
                           "transformers==4.57.6", "safetensors", "huggingface_hub>=0.25,<1",
                           "gradio>=4.44,<7", "sentencepiece", "protobuf"])
    if _before != _version("transformers") and "transformers" in sys.modules:
        raise SystemExit("Transformers was just changed to a compatible version, but the old one is still loaded. "
                         "Restart the session (Runtime → Restart session) and run this cell again.")

import torch
import gradio as gr
from transformers import (AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList,
                          TextIteratorStreamer)

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
STOP_EVENT = threading.Event()


def get_token():
    # token=None lets the Hub use an existing login; public downloads work without one.
    return HF_TOKEN.strip() or os.environ.get("HF_TOKEN") or None


def pick_dtype():
    if DTYPE != "auto":
        return {"float32": torch.float32, "bfloat16": torch.bfloat16, "float16": torch.float16}[DTYPE]
    if DEVICE == "cuda":
        return torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16
    return torch.float32


class StopOnEvent(StoppingCriteria):
    """Ends generate() as soon as the Stop button sets STOP_EVENT."""
    def __call__(self, input_ids, scores, **kwargs):
        return STOP_EVENT.is_set()


def load():
    token = None if os.path.isdir(MODEL_SOURCE) else get_token()
    dtype = pick_dtype()
    revision = None if os.path.isdir(MODEL_SOURCE) else MODEL_REVISION
    tok = AutoTokenizer.from_pretrained(MODEL_SOURCE, revision=revision, token=token, use_fast=True)
    if tok.pad_token_id is None:
        tok.pad_token = tok.eos_token
    model = AutoModelForCausalLM.from_pretrained(MODEL_SOURCE, revision=revision, token=token, trust_remote_code=True,
                                                 torch_dtype=dtype).to(DEVICE).eval()
    return model, tok, dtype


MODEL, TOK, MODEL_DTYPE = load()
CFG = MODEL.config
N_PARAMS = sum(p.numel() for p in MODEL.parameters())
CONTEXT = CFG.max_position_embeddings
print(f"Loaded {MODEL_SOURCE}: {N_PARAMS / 1e6:.1f}M params · {DEVICE} · {MODEL_DTYPE}")


def generate(prompt, max_new, temperature, top_k, top_p, rep_penalty, seed):
    STOP_EVENT.clear()
    if not prompt:
        yield "", "Type a prompt first."
        return
    ids = TOK(prompt, return_tensors="pt", add_special_tokens=False).to(DEVICE)
    room = CONTEXT - ids["input_ids"].shape[1]
    if room <= 0:
        yield prompt, f"Prompt is longer than the model's {CONTEXT}-token context; shorten it."
        return
    seed = int(seed)
    if seed >= 0:
        torch.manual_seed(seed)
    greedy = temperature <= 0
    sampling = {} if greedy else dict(temperature=float(temperature), top_k=int(top_k), top_p=float(top_p))
    streamer = TextIteratorStreamer(TOK, skip_prompt=False, skip_special_tokens=True, timeout=1.0)
    result = {}

    def worker():
        try:
            with torch.inference_mode():
                result["ids"] = MODEL.generate(
                    **ids, max_new_tokens=min(int(max_new), room), do_sample=not greedy,
                    repetition_penalty=float(rep_penalty), eos_token_id=TOK.eos_token_id, pad_token_id=TOK.pad_token_id,
                    use_cache=False, streamer=streamer, stopping_criteria=StoppingCriteriaList([StopOnEvent()]), **sampling)
        except Exception as exc:
            result["error"] = exc
            streamer.end()  # unblock the UI if the generation worker fails


    started = time.time()
    thread = threading.Thread(target=worker, daemon=True)
    thread.start()
    text = ""
    try:
        while True:
            try:
                piece = next(streamer)
            except queue.Empty:
                if not thread.is_alive():
                    break
                continue
            except StopIteration:
                break
            text += piece
            yield text, f"generating… {time.time() - started:.1f}s"
        thread.join()
    finally:
        if thread.is_alive():
            STOP_EVENT.set()
    if "error" in result:
        exc = result["error"]
        yield text or prompt, f"Generation failed: {type(exc).__name__}: {exc}"
        return
    new = result["ids"].shape[1] - ids["input_ids"].shape[1] if "ids" in result else 0
    took = time.time() - started
    status = "stopped" if STOP_EVENT.is_set() else "done"
    yield text or prompt, (f"**{status}** · {new} tokens · {took:.1f}s · {new / max(took, 1e-6):.1f} tok/s · "
                          f"{'greedy' if greedy else f'T={temperature} · top-k {int(top_k)} · top-p {top_p}'} · seed {seed}")


def output_box():
    kwargs = dict(label="Output (prompt + continuation)", lines=16, max_lines=30, interactive=False)
    for extra in (dict(show_copy_button=True), dict(buttons=["copy"]), {}):
        try:
            return gr.Textbox(**kwargs, **extra)
        except TypeError:
            continue


def build_ui():
    gr_major = int(gr.__version__.split(".")[0])
    theme = gr.themes.Soft(primary_hue="indigo", secondary_hue="blue")
    blocks_kwargs = {} if gr_major >= 6 else dict(theme=theme)     # Gradio 6 moved theme to launch()
    launch_kwargs = dict(theme=theme) if gr_major >= 6 else {}
    with gr.Blocks(title=MODEL_DISPLAY_NAME, **blocks_kwargs) as demo:
        gr.Markdown(f"# 🔥 {MODEL_DISPLAY_NAME}\n"
                    f"{N_PARAMS / 1e6:.1f}M-parameter math-first base model · `{CFG.num_hidden_layers}L × {CFG.hidden_size}d` · "
                    f"{CFG.num_attention_heads}Q/{CFG.num_key_value_heads}KV heads · context {CONTEXT} · "
                    f"continues text (not a chat model)")
        with gr.Row():
            with gr.Column(scale=3):
                prompt = gr.Textbox(label="Prompt", lines=6, placeholder="Write the beginning of the text to continue…")
                with gr.Row():
                    run = gr.Button("Generate", variant="primary")
                    stop = gr.Button("Stop", variant="stop")
                output = output_box()
                status = gr.Markdown("")
            with gr.Column(scale=1):
                max_new = gr.Slider(16, CONTEXT, value=min(DEF_MAX_NEW_TOKENS, CONTEXT), step=16, label="Max new tokens")
                temperature = gr.Slider(0.0, 1.5, value=DEF_TEMPERATURE, step=0.05, label="Temperature (0 = greedy)")
                top_k = gr.Slider(0, 200, value=DEF_TOP_K, step=1, label="Top-k (0 = off)")
                top_p = gr.Slider(0.05, 1.0, value=DEF_TOP_P, step=0.01, label="Top-p")
                rep = gr.Slider(1.0, 1.5, value=DEF_REPETITION_PENALTY, step=0.01, label="Repetition penalty")
                seed = gr.Number(value=DEF_SEED, precision=0, label="Seed (-1 = random)")
                with gr.Accordion("Model info", open=False):
                    gr.Markdown(f"- **Source:** `{MODEL_SOURCE}`\n- **Parameters:** `{N_PARAMS:,}`\n"
                                f"- **Device / dtype:** `{DEVICE}` / `{str(MODEL_DTYPE).replace('torch.', '')}`\n"
                                f"- **Vocab:** `{CFG.vocab_size:,}` · EOS `{TOK.eos_token}`")
        inputs = [prompt, max_new, temperature, top_k, top_p, rep, seed]
        events = dict(inputs=inputs, outputs=[output, status], concurrency_limit=1, concurrency_id="inferno-generation")
        run.click(generate, **events)
        prompt.submit(generate, **events)
        stop.click(lambda: STOP_EVENT.set(), queue=False)
    return demo, launch_kwargs


if __name__ == "__main__":
    demo, launch_kwargs = build_ui()
    demo.queue().launch(share=GRADIO_SHARE, server_port=GRADIO_PORT, server_name="0.0.0.0", **launch_kwargs)

Download the same script.

For this base model, write the beginning of the text you want continued, for example: Sam had 47 marbles. His sister gave him 28 more. Now Sam has. Set temperature to 0 for greedy decoding, or increase it for more varied stories.

Evaluation

All reported accuracies are percentages. Chance is 25% for ArithMark, HellaSwag and ARC; 50% for PIQA.

ArithMark 3.0

The official, unmodified AxiomicLabs scorer at commit 6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746 scores 1,000 four-choice continuation word problems. Settings: float32 weights, TF32 enabled as in the official entry point, fast training tokenizer, no added BOS token, batch size 8 examples (32 choices), max context 1,024, raw summed and per-token length-normalized log-likelihood. No few-shot examples.

Metric	Score
Normalized accuracy (primary)	85.70% (857/1000)
Raw accuracy	85.60% (856/1000)
Binomial standard error (normalized)	±1.11 points
Random-choice baseline	25.00%

Dataset SHA-256: bf8ab1a5193d52cdf0e05ff0b3ca226bdfcf416cb6e75562dcbe72e7e4559435. Full scores, predictions, category data and runtime settings are in evaluation/arithmark-3_results.json.

Per-category results

Category	N	Raw	Normalized
elementary_school_math_continuation::addition::grades_1_2::easy	128	90.62%	90.62%
elementary_school_math_continuation::comparison::grades_2_3::medium	44	100.00%	100.00%
elementary_school_math_continuation::comparison_difference::grades_2_3::medium	48	100.00%	100.00%
elementary_school_math_continuation::data::grades_2_3::easy	43	90.70%	90.70%
elementary_school_math_continuation::division::grades_3_4::medium	54	85.19%	85.19%
elementary_school_math_continuation::fractions_counting::grades_3_4::medium	50	58.00%	58.00%
elementary_school_math_continuation::geometry_area::grades_4_5::medium	52	100.00%	100.00%
elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium	45	80.00%	80.00%
elementary_school_math_continuation::measurement::grades_2_3::easy	76	96.05%	96.05%
elementary_school_math_continuation::money::grades_3_4::medium	64	100.00%	100.00%
elementary_school_math_continuation::multiplication::grades_3_4::medium	74	100.00%	100.00%
elementary_school_math_continuation::patterns::grades_3_4::medium	53	84.91%	84.91%
elementary_school_math_continuation::subtraction::grades_1_2::easy	117	77.78%	77.78%
elementary_school_math_continuation::time::grades_2_3::easy	55	100.00%	100.00%
elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium	46	34.78%	34.78%
elementary_school_math_continuation::two_step_addition::grades_2_3::medium	19	94.74%	94.74%
elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium	32	31.25%	34.38%

Language understanding

Source: evaluation/language_step_275000.json, selected from the trainer's lmeval_history.json at the released checkpoint's step. lm-evaluation-harness: zero-shot, full sets (limit=None), acc_norm, float32, batch size 8, max length 4096. Item counts below are the full task split sizes. Per-example predictions and harness standard errors were not retained by that training logger.

Task	Items	acc_norm
HellaSwag (validation)	10,042	33.68%
ARC-Easy (test)	2,376	51.60%
ARC-Challenge (test)	1,172	34.04%
PIQA (validation)	1,838	64.47%

To run a fresh evaluation of the published model:

lm_eval --model hf \
  --model_args "pretrained=AlgoDriveAI/Inferno-Exp-1.7,trust_remote_code=True,dtype=float32" \
  --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 8

Tokenizer reproducibility

The primary ArithMark result uses LlamaTokenizerFast from the exact saved training tokenizer. The earlier notebook runner described its requested tokenizer as “slow”; that label did not establish the actual tokenizer backend in its newer Transformers environment. During release verification under pinned Transformers 4.57.6, the fast tokenizer reproduced the reported 85.70% normalized / 85.60% raw result, including all 17 category scores. A genuine LlamaTokenizer slow-tokenizer check with the same weights gave 85.80% normalized / 86.00% raw. These are different tokenization paths, not different weights. The slow-tokenizer reference is retained in evaluation/arithmark-3_slow-tokenizer-reference.json and is not mixed into the primary result. In this release, the actual tokenizer class is recorded explicitly.

• training/train_inferno_exact.py: the user-supplied training source, byte-for-byte unchanged. Its historical INFERNO-EXP-1.6 header and default last.pt path are preserved.
• training/checkpoint_config.json: the actual configuration saved inside the released checkpoint, including inherited warm-start settings.
• training/checkpoint_provenance.json: checkpoint SHA-256, step, token progress, data identity and model-code fingerprint, where present.
• training/tokenizer/: original training tokenizer files, unchanged.
• evaluation/evaluate_arithmark3.py: standalone evaluation runner configured for this exact snapshot.
• release_manifest.json: SHA-256 and size for every release artifact.

Exact supplied training source SHA-256: ce792e3792a02edc3f31584294f4dfc68282d2f07f945168ec7ec905abf305f9. The source is supplied for reproducibility; its historical default settings are not rewritten to pretend they were the saved checkpoint configuration. The released inference weights omit optimizer/RNG state; the original full training checkpoint remains on Drive.

The root tokenizer uses the same vocabulary/EOS as training, with automatic BOS insertion disabled and BOS set to the training document EOS for tools that need a start-token ID, following the 1.6 export. The original tokenizer files remain under training/tokenizer/.

Limitations

This math-first base model can produce incorrect arithmetic and unreliable text. Its general language results are modest, and it is not trained to behave as a chat assistant. ArithMark was monitored during training and checkpoint selection, so its score is not an untouched holdout estimate. The four language tasks were also monitored during training.