Create README.md

README.md

@@ -0,0 +1,87 @@
+---
+language: en
+license: apache-2.0
+tags:
+- pretraining
+- small-language-model
+- educational
+- causal-lm
+datasets:
+- HuggingFaceTB/smollm-corpus
+---
+
+# Quark-50m-Instruct
+
+**Quark-50m-Instruct** is a causal (decoder-only) language model with approximately **50 million parameters**, trained from scratch on 5 billion tokens from the `smollm-corpus` dataset. It is designed to be lightweight, fast, and suitable for resource-constrained environments (e.g., RTX 3070, 8 GB VRAM), while retaining good text understanding and generation capabilities.
+
+The name *Quark* reflects its compact and elementary nature, ideal for on-device applications, lightweight conversational assistants, or as a base for domain-specific fine-tuning.
+
+## Model Details
+
+| Property               | Value                                |
+|------------------------|--------------------------------------|
+| Architecture           | Transformer decoder-only (SmolLM-style) |
+| Parameters             | ~50 M (effective, with weight tying) |
+| Context length         | 2048 tokens                          |
+| Vocabulary size        | 49,152 (cosmo2 tokenizer)            |
+| Model identifier       | `OvercastLab/Quark-50m-Instruct`     |
+| Primary language       | English (training data)              |
+
+## Architecture Details
+
+The model follows the style of **SmolLM** and **Qwen2.5** with the following characteristics:
+
+- **Grouped-Query Attention (GQA)** – ratio `n_heads / n_kv_heads = 3` to reduce KV cache footprint.
+- **SwiGLU** activation in feed-forward networks, with intermediate dimension `d_ff = 1024`.
+- **RMSNorm** applied before attention and FFN (pre-normalization).
+- **Rotary Positional Embeddings (RoPE)** with `theta = 10,000`.
+- **Weight tying** – input embedding and output projection share weights.
+- **Bias** – only on QKV projections (`qkv_bias = True`) for better numerical stability.
+
+| Component            | Configuration                         |
+|----------------------|---------------------------------------|
+| `d_model`            | 384                                   |
+| `n_layers`           | 24                                    |
+| `n_heads`            | 6                                     |
+| `n_kv_heads`         | 2                                     |
+| `head_dim`           | 64                                    |
+| `d_ff`               | 1024                                  |
+| `dropout`            | 0.0 (no dropout during pretraining)   |
+
+## Training Data
+
+The model was pretrained on **5 billion tokens** sampled from the [`HuggingFaceTB/smollm-corpus`](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) dataset with the following distribution:
+
+| Sub-dataset               | Percentage | Tokens (billions) | Main content                                 |
+|---------------------------|------------|-------------------|----------------------------------------------|
+| `cosmopedia-v2`           | 60%        | 3.0               | Synthetic textbooks, educational articles, stories |
+| `fineweb-edu-dedup`       | 40%        | 2.0               | Web pages filtered for educational quality   |
+
+Data were tokenized using the `HuggingFaceTB/cosmo2-tokenizer` (vocabulary size 49,152), with the EOS token appended to each document. Training sequences have a fixed length of **2048** tokens (with packing).
+
+## Training Procedure
+
+- **Framework**: PyTorch with `torch.compile` and `GradScaler` for mixed precision.
+- **Precision**: `bfloat16` (Ampere RTX 3070).
+- **Optimizer**: AdamW (`β₁=0.9`, `β₂=0.95`, weight decay = 0.1).
+- **Learning rate**: `3e-4` with linear warmup for 1,000 steps, then cosine decay to `3e-5`.
+- **Effective batch size**: 64 sequences × 2048 tokens = **131,072 tokens per step**.
+  - Micro-batch: 4 sequences, gradient accumulation over 16 steps.
+- **Gradient clipping**: 1.0.
+- **Total steps**: approximately 38,000 (to reach 5B tokens).
+
+## Usage
+
+You can load and use the model directly with the `transformers` library:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "OvercastLab/Quark-50m-Instruct"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+
+input_text = "The theory of relativity"
+inputs = tokenizer(input_text, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=100)
+print(tokenizer.decode(outputs[0]))