SnapKitty Transformer 8B

8B Parameter Transformer Model

A sovereign corporate 8B parameter transformer model with production-ready FlashAttention CUDA kernels, Triton reference implementations, MoE routing, and YaRN RoPE — optimized for NVIDIA Ampere (sm_80+) and Hopper architectures.

⚠️ NOT OPEN SOURCE — This is a corporate sovereign product. Commercial use requires a Sovereign Node Key.

Topics: transformer 8b flashattention cuda nvidia tensor-cores attention-mechanism language-model sovereign copyleft bsl agpl sm_80 hopper moe triton yarn mxfp4

Architecture

Model Specifications

Parameter	Value
Total Parameters	8B
Hidden Dimension	8192
Attention Heads	128
Key-Value Heads	16 (GQA)
Head Dimension	64
Layers	128
Vocabulary Size	256,000
Max Sequence Length	204,800
Experts	8 (MoE, top-4 routing)
Sliding Window	65,536
Attention Sinks	2,048
YaRN RoPE	32x factor
MXFP4 Quantization	Supported
Precision	FP16 / BF16 / MXFP4

FlashAttention Kernel

Production-ready fused GEMM + online softmax kernel for NVIDIA Ampere (sm_80+):

cp.async: Asynchronous global-to-shared memory copies
mma.sync.aligned.m16n8k16: Tensor Core matrix multiply-accumulate
Online softmax: Running max/sum accumulators without materializing full attention matrix
Double buffering: Overlapping compute with memory loads
Warp-level reductions: __shfl_down_sync for efficient reductions

// Kernel Configuration
constexpr int Br = 64;       // Rows of Q per block
constexpr int Bc = 64;       // Rows of K/V per tile
constexpr int d = 64;        // Head dimension
constexpr int num_warps = 4; // 128 threads total

Repository Structure

snapkitty-transformer/
├── kernels/                          CUDA & Triton kernels
│   ├── flash_attention_fwd.cu        FlashAttention forward (Ampere sm_80+)
│   ├── flash_attention_bwd.cu        FlashAttention backward pass
│   ├── flash_attention_hopper.cu     FlashAttention-3 forward (Hopper, TMA, FP8, cluster MMA)
│   ├── flash_attention_hopper_bwd.cu FlashAttention-3 backward (Hopper, dQ/dK/dV, atomic reduce)
│   ├── flash_attention.cpp           PyTorch C++ extension
│   ├── mxfp4_quant.cu               MXFP4 quant/dequant (E2M1, block scales, MoE layout)
│   ├── attention_kernel.py           Triton FlashAttention (sliding window + sinks)
│   └── moe_kernel.py                 Triton fused MoE (MXFP4, top-4, SwiGLU, TP)
├── gpt_oss/                          Model architecture
│   └── model.py                      GPT-OSS: RMSNorm, YaRN RoPE, SwiGLU, MoE, Attention
├── benchmark.py                      FlashAttention benchmark suite
├── analyze_ncu.py                    NCU log parser & bottleneck analysis
├── flash_attention_triton.py         Triton reference FlashAttention
├── setup.py                          Build config (PyTorch C++ extension)
├── test_flash_attention.py           Forward tests
├── test_hopper_bwd.py                Backward + MXFP4 round-trip tests
├── LICENSE                           Business Source License 1.1
├── LICENSE-AGPL                      GNU AGPL v3.0
└── README.md                         This file

Quick Start

Installation

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers datasets accelerate triton

Build CUDA Extension

cd snapkitty-transformer
python setup.py install

Model Loading

from gpt_oss.model import GPTOSSModel, ModelConfig

config = ModelConfig(
    vocab_size=256000,
    dim=8192,
    n_heads=128,
    n_kv_heads=16,
    n_layers=128,
    max_seq_len=204800,
    num_experts=8,
    top_k=4,
)

model = GPTOSSModel(config)
model.load_state_dict(torch.load("snapkitty-8b-weights.pt"))
model = model.cuda().half()

FlashAttention Inference

from kernels.flash_attention import flash_attention_fwd

# Q, K, V: [batch, heads, seq_len, head_dim]
output = flash_attention_fwd(Q, K, V, causal=True, sliding_window=65536, num_sinks=2048)

Text Generation

input_ids = tokenizer("The sovereign substrate is", return_tensors="pt").input_ids.cuda()
output = model.generate(input_ids, max_new_tokens=512, temperature=0.8)
print(tokenizer.decode(output[0]))

Training

Pre-training Data

2T tokens from curated sovereign datasets
Code: 500B tokens (Python, Rust, CUDA, Solidity)
Academic: 300B tokens (arXiv, textbooks)
Web: 1.2T tokens (filtered, deduplicated)

Training Configuration

model:
  vocab_size: 256000
  dim: 8192
  n_layers: 128
  n_heads: 128
  n_kv_heads: 16
  head_dim: 64
  max_seq_len: 204800
  num_experts: 8
  top_k: 4
  expert_intermediate_dim: 2048
  sliding_window: 65536
  num_sinks: 2048
  rope_yarn: true
  rope_yarn_factor: 32.0
  use_mxfp4: true

training:
  batch_size: 2048
  learning_rate: 3e-4
  warmup_steps: 2000
  total_steps: 500000
  optimizer: AdamW
  weight_decay: 0.1
  grad_clip: 1.0
  bf16: true

Performance

Benchmarks (A100 80GB)

Metric	Value
Inference (FP16)	45 tokens/sec
Inference (INT8)	62 tokens/sec
Inference (INT4)	78 tokens/sec
Training Throughput	180 tokens/sec/GPU
FlashAttention Speedup	2.5x vs naive

Memory Usage

Precision	Model Size	KV Cache (8K)
FP16	16 GB	2 GB
INT8	8 GB	1 GB
INT4	4 GB	0.5 GB

Protected Inventions

SNAPKITTY TRANSFORMER ARCHITECTURE 8B parameter transformer with 128 layers, 128 attention heads, 16 GQA KV heads, 8-expert MoE with top-4 routing, and 204800 max sequence length.
FLASHATTENTION FUSED GEMM + ONLINE SOFTMAX KERNEL Production-ready CUDA kernel with cp.async, mma.sync, and double-buffered K/V tile processing.
FLASHATTENTION-3 HOPPER FORWARD KERNEL TMA-accelerated, warp-specialized kernel with cluster MMA, K-stage pipelining, FP8 computation, causal masking, sliding window, attention sinks, and YaRN RoPE integration.
FLASHATTENTION-3 HOPPER BACKWARD KERNEL Full dQ/dK/dV computation with S/P recomputation from L/M, dO@V^T accumulation, dS=P*(dO@V^T-D), atomic partial reduce across Q tiles, and global reduction kernel.
TRITON FLASHATTENTION WITH SLIDING WINDOW + SINKS Reference Triton implementation with online softmax, sliding window attention, and persistent attention sinks.
TRITON FUSED MoE KERNEL Triton Mixture-of-Experts with MXFP4 dequantization, top-4 routing, SwiGLU activation, and tensor parallelism support.
YARN ROTARY POSITIONAL EMBEDDING YaRN-scaled RoPE with 32x factor for 204800-length sequences, integrated into both CUDA and Triton kernels.
MXFP4 QUANTIZATION/DEQUANTIZATION E2M1 4-bit format with shared block exponent (E8M0), 32-element blocks, LUT-based rounding, CPU reference + device kernels, PyTorch bindings, and MoE weight layout for kernel consumption.
GPT-OSS MODEL ARCHITECTURE Complete PyTorch module with RMSNorm, YaRN RoPE, SwiGLU, MXFP4 dequantization, sliding window + sinks attention, MoE with top-4 routing, and full generation pipeline.
SOVEREIGN TRAINING DATA PIPELINE Proprietary data curation, filtering, and augmentation methods.
INFERENCE OPTIMIZATION SUITE Quantization, pruning, and distillation techniques.

License

⚠️ THIS IS NOT OPEN SOURCE

This project is a sovereign corporate product licensed under Business Source License 1.1 (BSL-1.1) with GNU AGPL v3.0 copyleft for network services.

License Structure:

Component	License	File	Scope
Core Model & Kernels	BSL-1.1 → AGPL v3.0 (2030)	`LICENSE`	Transformer architecture, CUDA kernels, training pipeline
API/Server	GNU AGPL v3.0	`LICENSE-AGPL`	REST API, WebSocket server, inference endpoints

Sovereign Node Key Required:

Commercial use: Requires Sovereign Node Key from Licensor
Network service: AGPL v3.0 copyleft triggers
Derivative works: Must be licensed under same BSL-1.1 terms
No relicensing: Without explicit written consent from Licensor

CopyLeft Provision:

Any derivative work based on the Licensed Work must be licensed under the same Business Source License 1.1 terms. No relicensing permitted without explicit written consent from the Licensor.

Citation

@misc{snapkittytransformer2026,
  title={SnapKitty Transformer 8B},
  author={Ahmad Ali Parr and Jessica Westerhoff},
  year={2026},
  note={8B parameter transformer with FlashAttention CUDA kernels},
  publisher={SNAPKITTYWEST},
  howpublished={\url{https://github.com/SNAPKITTYWEST/snapkitty-transformer}},
  keywords={transformer, 8b, flashattention, cuda, nvidia, tensor-cores, sovereign},
  license={BSL-1.1}
}

Contact

Ahmad Ali Parr - ahmedparr93@gmail.com Jessica Westerhoff - jessicalw34@gmail.com

Bel Esprit d'Accord Trust — 50/50 equal sovereigns

The substrate is not for sale. It is not for porting. It is for Execution in the Wild.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support