Qyrou-1 65M · Qyrou-1 Vega
We are introducing the experimental release of Qyrou-1 — a family of Small Language Models (SLMs) designed for practical, efficient parameter sizes. The series features three distinct variants: Sao (35M), Vega (65M), and Aeon (125M). This model represents an experimental checkpoint from the initial pre-training phase. It is provided as-is for custom fine-tuning and further research experiments.
The 35M and 125M are set to be released once the 65M base model is fully trained. As of now, the model has been trained on a diverse corpus that contains mostly English (for language representations), as well as Math, Grade school science, basic coding, and step-by-step reasoning samples, which break down complex tasks into smaller subgroups, although not on the full, curated dataset.
The Qyrou-1 series is built for researchers and developers exploring Small Language Models (SLMs), novel architecture designs, and practical model integration. It is also designed to support intelligent features in applications such as the upcoming QyrouCode release (not as an agent model) once fine-tuned!
Qyrou-Vega models are the mid-sized variants within the Qyrou model family. They are designed to handle a broad range of tasks typically supported by small language models, including summarization, fill-in-the-middle (FIM), basic code completion, text generation, mathematical reasoning, and more.
Quickstart
This model is compatible with standard Hugging Face Transformers workflows and may be used as a remote-code model. Before proceeding, install the required Python dependencies. An NVIDIA GPU with a recent driver is strongly recommended for optimal performance; additional guidance is provided in the GPU acceleration section below.
pip install --upgrade transformers torch tokenizers
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Qyrou/Qyrou-1-EXP-Base", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Qyrou/Qyrou-1-EXP-Base")
model = model.to("cuda").to(torch.bfloat16) if torch.cuda.is_available() else model.eval()
inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Generation is supported through the standard GenerationMixin interface, including greedy decoding, sampling, beam search, and batch decoding with padding. The hybrid incremental cache, which combines attention key/value states with short-convolution state, is created and updated automatically; as a result, no explicit cache management is required on the caller side. Generation with use_cache=True is functionally equivalent to a full forward pass, including for right-padded and unequal-length batches.
The serving contract follows standard Transformers conventions. The model accepts output_hidden_states, output_attentions, attention_mask, position_ids, past_key_values, cache_position, use_cache, and the standard framework arguments. Attention weights are returned for every GQA layer when output_attentions=True, and hidden states are returned after the embedding layer and after each block. Unsupported arguments are rejected with clear errors rather than being silently ignored, which helps surface integration issues early.
Model Architecture
| Component | Qyrou specification |
|---|---|
| Model type | Qyrou; decoder-only hybrid causal language model |
| Total parameters | 65,304,192 (~65.3M) |
| Vocabulary | 20,000 tokens |
| Context length | 2,048 tokens |
| Hidden size | 512 |
| Blocks | 21 |
| Normalization | Pre-RMSNorm |
| Query/key states | QK-norm (per-head RMSNorm on query and key states) |
| Positional encoding | RoPE (θ = 10,000) |
| Token embeddings / LM head | Tied |
| Attention blocks | 17 |
| Attention type | Causal grouped-query attention (GQA) |
| Query heads | 8 |
| Key/value heads | 2 |
| Query groups per KV head | 4 |
| Head dimension | 64 |
| Convolution blocks | 4 |
| Convolution layers | 4, 9, 14, 19 |
| Convolution type | Causal depthwise Conv1D + pointwise projection |
| Convolution kernel size | 7 |
| Feed-forward network | SwiGLU in every block |
| FFN dimensions | 512 → 1,328 → 512 |
| Activation | SiLU |
| Biases | None |
| Per-block structure | x = x + Mixer(RMSNorm(x)) → x = x + SwiGLU_FFN(RMSNorm(x)) |
Qyrou-Vega is a 65.3M-parameter language model built around the qyrou-arch architecture. It was trained with a 2,048-token context window, although it could be extended beyond that with updated RoPE settings and cache support; there is no confirmed maximum yet (Although training beyond a sequence length of 2k could degrade a 65M SLM). The model has 21 layers: 17 use grouped-query attention, while 4 use causal convolutions in layers 4, 9, 14, and 19. Each layer uses RMSNorm and SwiGLU, with a hidden size of 512, a 20,000-token vocabulary, eight attention heads, two key/value heads, and tied input and output embeddings. Query and key states are normalized with per-head RMSNorm (QK-norm) before the rotary embedding is applied.
Tokenizer
Qyrou-1-EXP uses a flexible, extensible byte-level BPE tokenizer designed to support continued experimentation across a broad range of workloads. The tokenizer is suitable for:
- General natural-language processing
- Language understanding and reasoning
- Conversational and instruction-following tasks
- ChatML-formatted interactions
- Code completion and Fill-in-the-Middle (FIM)
- Mathematical and symbolic reasoning
- Vision-, audio-, and multimodal token integration
- Tool-use and structured interaction formats
The included tokenizer is ready for immediate use with the model and provides a stable foundation for fine-tuning, evaluation, and experimentation across these domains. It uses a 20,000-token vocabulary and preserves byte-level coverage, enabling robust handling of Unicode text, source code, mathematical notation, formatting, and previously unseen text without relying on an unknown-token fallback in normal usage.
The tokenizer also includes a broad set of reserved and atomic control markers. These cover common conventions used by contemporary language models, including ChatML, reasoning boundaries, tool calls, FIM workflows, multimodal placeholders, vision and audio boundaries, and reserved extension tokens. Markers are encoded atomically to ensure that control structures remain stable and unambiguous during training and inference.
Core token IDs are fixed for compatibility:
| Token | ID |
|---|---|
<|bos|> |
0 |
<|eos|> |
1 |
<|pad|> |
2 |
<|unk|> |
3 |
<|eod|> |
4 |
The tokenizer uses a byte-level pre-tokenizer and decoder, allowing lossless round-trip encoding for supported text. It is configured with a 2,048-token model context length and includes a ChatML-compatible chat template using <|im_start|>, role markers such as <|user|> and <|assistant|>, and <|im_end|> delimiters.
Some control markers are intentionally represented as atomic regular-text tokens rather than removable special tokens. This allows fine-tuning workflows to preserve reasoning, tool-use, FIM, and multimodal markup when required. Applications should therefore select skip_special_tokens deliberately when decoding model outputs.
Users are free to fine-tune Qyrou-1-EXP for their own tasks, including domains that require additional control symbols or structured token sequences. If a required token or marker is not currently available, please open a discussion in this model repository describing the intended use case and the proposed token. Feedback from these experiments will be considered for future tokenizer revisions and subsequent model versions.
Data Used For Training
Note: The benchmark datasets listed here are intended for evaluation. For any datasets used during training, only the official training split was utilized.
Nvidia GPU Acceleration
We built this experimental model to run best on NVIDIA hardware, so use an NVIDIA GPU whenever possible. For maximum performance, enable BF16, cuDNN or Flash Attention, torch.compile, fused QKV and SwiGLU projections, and the included custom Triton kernels. Install Liger Kernel and Cut Cross-Entropy to enable the additional optimized paths. Together, these features reduce memory usage, kernel-launch overhead, and unnecessary computation while taking advantage of NVIDIA Tensor Cores.
llama.cpp support will take some time. I'm focusing on getting the architecture working correctly first, then I'll roll out performance optimizations gradually as the full model family is released.
Development
The repository is organized in a relatively straightforward manner. The implementation is contained in the qyrou_arch/ directory, the test suite is located in the tests/ directory, and the standalone utilities are stored in the scripts/ directory. The test suite may be executed with pytest from the source/ directory. It evaluates the core behavior of the model, including numerical parity for each layer against manual references, cached and uncached forward passes (including left-padded, right-padded, and uneven batch sizes), the serving interface, and the CUDA-gated Triton kernel paths. The CPU-based tests are expected to pass directly, while the Triton-specific tests assess FP32 and BF16 behavior, edge cases such as tail widths and non-contiguous inputs, and larger GPU batch sizes.
The scripts/benchmark_micro.py script functions as a lightweight benchmarking utility for the primary kernels, including packed QKV and SwiGLU GEMMs, RMSNorm, SDPA backends, and the cross-entropy path. It produces a JSON report containing median timings, which is useful for identifying regressions across different hardware configurations. CUDA is required to run this script.
The Git history may contain files such as
attributions.md. Please disregard this file, as it contains a contaminated list of datasets from a separate project and is not associated with Qyrou-1-EXP. The datasets relevant to this project are documented in this README. Any other files identified as contaminated have been removed. We appreciate your understanding, as maintaining and managing open-source projects can involve significant challenges. If you encounter any bugs or other issues with the repository, please open a discussion in the repository to report them!
Limitations
Qyrou-Vega was pretrained on a diverse dataset; however, as a small language model (SLM), it does not match the capabilities or performance of larger models trained on substantially larger and more diverse corpora. This release represents the first-stage pretraining (PT) run and is provided strictly for experimental and research purposes. As the first experimental base model in the Qyrou-1 family, it is intended primarily for experimentation and fine-tuning and is not recommended for production use until an official stable release is released by the Qyrou organization. Furthermore, Qyrou-Vega is a base model and has not been instruction-tuned, meaning it is not optimized to follow user instructions or structured response formats and may produce inconsistent or unstable outputs until properly fine-tuned. By using this model, you acknowledge its experimental nature and assume full responsibility for its use. The Qyrou organization and its contributors are not liable for any damages, losses, or consequences arising from the use or misuse of this model.
- Downloads last month
- -