Upload 8 files

README.md

@@ -1,139 +1,57 @@
 ---
-language:
-- en
-- code
-license: apache-2.0
+library_name: transformers
+model_name: sft_id
 tags:
-- smol
-- pretraining
-- instruct
-- 50M
-- causal-lm
-- gqa
-- swiglu
-- rmsnorm
-datasets:
-- HuggingFaceTB/smollm-corpus
-metrics:
-- perplexity
-model-index:
-- name: Quark-50m-Instruct
-  results: []
-pipeline_tag: text-generation
+- generated_from_trainer
+- trl
+- sft
+licence: license
 ---
 
-# Quark-50m-Instruct
+# Model Card for sft_id
 
-**Quark-50m-Instruct** is a small (≈50M parameters) decoder-only language model, fine-tuned for instruction following.
-It is built on the same architecture as the now‑abandoned “SmolLM” family and was fully pretrained on 5 billion tokens from
-[HuggingFaceTB/smollm‑corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus).
+This model is a fine-tuned version of [None](https://huggingface.co/None).
+It has been trained using [TRL](https://github.com/huggingface/trl).
 
-- **Model type:** Causal Language Model (LLaMA‑style decoder)
-- **Architecture:** GQA · SwiGLU · RMSNorm · RoPE · Weight‑tying
-- **Pretraining tokens:** 5 B
-- **Fine‑tuning:** Instruction‑tuned (details below)
-- **Creators:** [OvercastLab](https://huggingface.co/OvercastLab) (research & development lab for ML/AI)
-- **Release date:** 22 April 2026
+## Quick start
 
-## Model Summary
-
-Quark-50m-Instruct is designed to be an efficient assistant that can run on consumer GPUs (e.g., RTX 3070 with 8 GB VRAM)
-and even on CPU for light workloads. It is **not** competitive with large models on knowledge‑intensive tasks,
-but it excels at:
-
-- Simple conversational tasks
-- Code generation and explanation (Python)
-- Short text rewriting and summarisation
-- On‑device / edge inference
-
-The architecture closely follows the efficient‑small‑LM blueprint popularised by SmolLM:
-
-| Component   | Details                       |
-|-------------|-------------------------------|
-| Vocab size  | 49,152                        |
-| Hidden size | 384                           |
-| Layers      | 24                            |
-| Attention   | Grouped Query (6 Q heads, 2 KV heads) |
-| FFN         | SwiGLU with 1,024 intermediate |
-| Position    | RoPE (θ = 10,000)             |
-| Normalisation | RMSNorm (pre‑block)         |
-
-Total trainable parameters: **≈48 M** (with weight tying).
-
-## Uses
-
-### Direct Use
-The model can be used via the 🤗 Transformers library for standard text generation.
-It expects chat‑formatted input (see example below).
-
-### Downstream Use
-Because of the open Apache‑2.0 license, you may fine‑tune Quark-50m‑Instruct on your own data for
-domain‑specific tasks – for instance, a customer‑support bot, a code reviewer, or a story writer.
-
-### Limitations
-- Limited world knowledge (stopped at mid‑2025 pretraining data).
-- Short context window (2,048 tokens).
-- Small size means it can make more factual mistakes than larger models.
-
-## Training Details
-
-### Pretraining
-
-The base model was pretrained from scratch on a single NVIDIA A100.
-Training took approximately **One Day**.
-
-#### Data mix
-
-Quark‑50m was trained on exactly 5 billion tokens sampled from `HuggingFaceTB/smollm-corpus` with the following proportions:
-
-| Subset            | Share | Tokens |
-|-------------------|-------|--------|
-| cosmopedia‑v2     | 60%   | 3.0 B  |
-| fineweb‑edu‑dedup | 40%   | 2.0 B  |
+```python
+from transformers import pipeline
 
-All data was tokenised with the official [Cosmo2 tokenizer](https://huggingface.co/HuggingFaceTB/cosmo2-tokenizer) (vocab size 49,152).
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
 
-#### Hyperparameters (pretraining)
+## Training procedure
 
-| Parameter               | Value                      |
-|-------------------------|----------------------------|
-| Sequence length         | 2,048                      |
-| Micro‑batch size        | 4                          |
-| Gradient accumulation   | 16                         |
-| Effective batch         | 64 seqs (≈131k tokens)     |
-| Optimizer               | AdamW (β₁=0.9, β₂=0.95)   |
-| Learning rate           | 3e‑4 → 3e‑5 (cosine decay)|
-| Warmup steps            | 1,000                      |
-| Weight decay            | 0.1                        |
-| Gradient clipping       | 1.0                        |
-| Mixed precision         | bfloat16                   |
+ 
 
-### Instruction Fine‑tuning
 
-The base model was fine‑tuned on a curated set of instruction‑following data (details to be released).  
-The fine‑tuning used **LoRA** with the same sequence length and a lower learning rate (1e‑4) for a few thousand steps.
 
-## How to Use
+This model was trained with SFT.
 
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
+### Framework versions
 
-model_name = "OvercastLab/Quark-50m-Instruct"
+- TRL: 1.2.0
+- Transformers: 5.6.2
+- Pytorch: 2.4.1+cu124
+- Datasets: 4.8.4
+- Tokenizers: 0.22.2
 
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
+## Citations
 
-messages = [
-    {"role": "system", "content": "You are Quark, a helpful assistant."},
-    {"role": "user", "content": "Explain group query attention in one sentence."}
-]
 
-inputs = tokenizer.apply_chat_template(
-    messages,
-    tokenize=True,
-    add_generation_prompt=True,
-    return_tensors="pt"
-).to(model.device)
 
-outputs = model.generate(inputs, max_new_tokens=128)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+Cite TRL as:
+    
+```bibtex
+@software{vonwerra2020trl,
+  title   = {{TRL: Transformers Reinforcement Learning}},
+  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
+  license = {Apache-2.0},
+  url     = {https://github.com/huggingface/trl},
+  year    = {2020}
+}
+```

config.json

@@ -28,7 +28,7 @@
     "rope_type": "default"
   },
   "tie_word_embeddings": true,
-  "transformers_version": "5.6.1",
+  "transformers_version": "5.6.2",
   "use_cache": false,
   "vocab_size": 49152
 }

generation_config.json

@@ -6,6 +6,6 @@
     2
   ],
   "pad_token_id": 0,
-  "transformers_version": "5.6.1",
+  "transformers_version": "5.6.2",
   "use_cache": true
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:431dbd275b83cb41bd28cdd1bb6d9c30e87ed1c5da31957e70867c9cc095efa7
+oid sha256:ae5952eb75980cb4d0a309de77867170af85f62c992c16db60c244ad71798cd6
 size 113367352

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7111b6742ad6cc5ab900057295f3d9c66f5ee720c6c73b73b0f9abad6b7f195c
+oid sha256:82a7ba29d21c786c5562de6a6c7a3aa158352399f3651c4038d36d59bf4bd17e
 size 5304