arist999/gelocity-alpha-3.5

Gelocity Alpha 3.5

AI Coding Assistant built from scratch with Python + PyTorch.

Model Architecture

• Transformer with GQA (Grouped Query Attention)
• RoPE (Rotary Position Embedding)
• RMSNorm
• SwiGLU activation
• Mixture of Experts (MoE) with top-k routing
• FlashAttention CPU fallback

Usage

from model import SycalModel
import torch

# Load model
model = SycalModel(vocab_size=300, dim=32, n_layers=2, n_heads=2, n_kv_heads=1)
model.eval()

# Generate (placeholder - needs training)
# TODO: Implement generation after training

Training Status

• Phase: Pre-training (smoke test completed)
• Parameters: 0.04M (smoke test), planned 100M-1B
• Checkpoint: Available after full training

Roadmap

• Model architecture
• Tokenizer (BPE)
• Data pipeline
• Training loop (smoke test)
• Scale-up training (needs GPU)
• RL fine-tuning (GRPO)
• Evaluation on code tasks

Hardware

• Training: CPU AVX2 + OpenMP (smoke test)
• Production: Needs GPU NVIDIA for full training

License

MIT License

Contact

Created by Gelocity Team