Sm0r3s/CryogenNet-151M

CryogenAI 151M

CryogenAI 151M is the custom neural language-model core of the CryogenNet project.

It is a recurrent decoder-only architecture implemented in PyTorch rather than a standard Llama/Qwen/Mistral model.

Architecture CryogenNet

Property	Value
Parameters	151,005,697
Vocabulary	24,576
Hidden size	1,536
Attention heads	24
FFN size	6,144
Shared recurrent blocks	2
Coda blocks	1
Maximum architecture context	4,096
Base-training context	2,048
Training recurrence	R=2–4
Position encoding	RoPE
Normalization	RMSNorm
FFN	SwiGLU
Framework	PyTorch

Structure

text
 ↓
CryogenTokenizer
 ↓
token embeddings
 ↓
┌──────────────────────────┐
│ shared recurrent core    │
│       repeated R times   │
└──────────────────────────┘
 ↓
coda block
 ↓
RMSNorm
 ↓
tied LM head
 ↓
logits

The recurrent depth R can be varied during inference.

Tokenizer

CryogenTokenizer uses SentencePiece BPE with:

• 24,576-token vocabulary
• byte fallback
• NFC normalization
• case preservation
• whitespace/code-indentation preservation
• Cryogen-native control tokens

Tokenizer SHA-256:

6b776cb9a9d9760de07dfb670bf5fff19fa879e271301a7dbd493b024c938f34

Base pretraining

The base model was trained on CryogenCorpus v1, targeting:

249,429,568 training tokens

The corpus contains general text, code, mathematics, scientific/technical material, reasoning data, and structured/tool-use examples.

Loading in PyTorch

import torch
from modeling_cryogennet import CryogenNet

model = CryogenNet(
    vocab_size=24576,
    dim=1536,
    n_heads=24,
    ffn_dim=6144,
    n_shared_blocks=2,
    n_coda_blocks=1,
    dropout=0.0,
    max_seq_len=4096,
)

state = torch.load(
    "cryogennet-151m-fp16.pt",
    map_location="cpu",
)

model.load_state_dict(state)
model.eval()

CryogenNet currently requires the included custom PyTorch architecture code. It is not directly compatible with Ollama, llama.cpp, LM Studio, or standard AutoModelForCausalLM loading without a dedicated CryogenNet backend.

CryogenAI roadmap

CryogenNet is the base neural core. Later CryogenAI stages may add:

• CryoReasoning fine-tuning
• verifier/confidence components
• knowledge integration
• tool routing
• optimized KV-cached inference
• quantized/mobile deployment

Status

Experimental research model.