\u6c38 Continual AI 2.0B Foundation Checkpoint
Continual AI is a 2.0 Billion parameter decoder-only Continual Transformer engineered for lifelong continuous learning without catastrophic forgetting.
- Live Interactive Space: Vir007/continual-ai
- Live Web App: https://vir007-continual-ai.static.hf.space/index.html
\ud83d\udcd0 Architecture Specifications
- Total Parameters: ~2.0 Billion
- Layers: 28 transformer blocks
- Hidden Dimension ({\text{model}}$): 2560
- Attention Heads ({\text{heads}}$): 20
- KV Heads ({\text{kv\_heads}}$): 4 (Grouped Query Attention 5:1 ratio)
- Head Dimension ({\text{head}}$): 128
- Intermediate Dimension (SwiGLU): 6656
- Vocabulary Size: 50,257 tokens
- Context Window: 512 tokens (extendable with RoPE theta scaling)
- Position Embeddings: Rotary Position Embeddings (RoPE, $\theta = 10000.0$)
- Normalization: RMSNorm ($\epsilon = 1e-5$)
- Attention: FlashAttention-2 / SDPA via orch.nn.functional.scaled_dot_product_attention
- Precision: Native BF16 / FP16 mixed precision
\ud83e\udde0 Continual Learning & Zero-Forgetting Mechanics
- Dual-System Plasticity:
- Fast System (Episodic & LoRA): Low-rank adapters (stage4_2b_lora.pt) handle rapid online updates with fast-trace synaptic bounds.
- Slow System (Base Weights): Deep foundation weights (stage4_2b.pt) preserve core language representations.
- Dynamic Mixture-of-Experts (MoE):
- Routing gates (stage4_2b_moe.pt) partition representations into specialized domain clusters with entropy regularization.
- Autonomous Sleep & Dreaming Consolidation:
- Background synaptic replay dissipates energy ($\Delta E$) and consolidates episodic memories into structural adapters.
- Verified anchor drift: 0.0000 bits across sequential task benchmarks.
- Cloud Lineage Tracking:
- User-independent mathematical digests tracked via Convex (dependable-swordfish-420).
\ud83d\udce6 Checkpoint Files Included
| File | Size | Description |
|---|---|---|
| stage4_2b.pt | ~3.97 GB | Base 2.0B Transformer weights (PyTorch state dict) |
| stage4_2b_lora.pt | ~12 MB | Continuous plastic LoRA adapter weights |
| stage4_2b_moe.pt | ~12 MB | Dynamic MoE routing gates and expert adapters |
| config.json | ~1 KB | Complete architectural hyperparameter configuration |
\ud83d\udcbb How to Load in PyTorch
`python import torch from continual_ai.gpu_transformer import TransformerConfig, ScaledContinualTransformer
1. Initialize 2.0B Architecture Configuration
config = TransformerConfig.stage4_2b()
2. Instantiate Model
device = 'cuda' if torch.cuda.is_available() else 'cpu' dtype = torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float32
model = ScaledContinualTransformer(config).to(device=device, dtype=dtype)
3. Load Checkpoint Weights
checkpoint_path = 'stage4_2b.pt' state_dict = torch.load(checkpoint_path, map_location=device) model.load_state_dict(state_dict, strict=False) model.eval()
print('Continual AI 2.0B model loaded successfully!') `
\ud83d\udcdc Citation & License
- License: MIT
- Author: Vir007 / Continual AI Architecture
- Downloads last month
- -