TS-A
Rose X2 SLM (167.3M params).
Training Progress
- Tokens Seen: 11,319,091,200
- Compute Tokens: 11,319,091,200
- Step: 21,600
- Val PPL: inf
Architecture
| Property | Value |
|---|---|
| Layers | 30 |
| Hidden | 640 |
| Heads | 10 (kv=5) |
| Diff Attention | True |
| Logit Softcap | True (cap=50.0) |
| Params | 167.258M |
Features
- GQA
- QK-Norm
- RoPE
- SwiGLU
- RMSNorm
- DifferentialAttention
- LogitSoftcap
- XSA
- RefreshGate
- TST
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support