Update README.md
Browse files🧙♂️ LLightPro
Small model. Massive logic.
This is a high-fidelity fine-tune of the experimental p-e-w/Qwen3-4B-Instruct-2507-heretic base model, sharpened for complex reasoning, coding, and logic puzzles using the elite GrandMaster2 dataset.
Unlike standard LoRA fine-tunes, this model uses DoRA (Weight-Decomposed Low-Rank Adaptation), allowing it to learn subtle nuances in reasoning without catastrophic forgetting. The training was performed in pure bfloat16 (no quantization during training) on an NVIDIA RTX 4090 to ensure maximum precision.
🚀 Key Features
🧠 Advanced Architecture: Built upon the experimental "Heretic" Qwen3 build.
⚡ DoRA Technology: Uses Weight-Decomposed LoRA (r=64, alpha=128) for superior learning capacity compared to standard LoRA.
💎 Uncompromised Quality: Trained in native bfloat16 precision. No 4-bit or 8-bit quantization was used during the backpropagation process.
📚 Elite Data: Fine-tuned on an optimized version of Vikhrmodels/GrandMaster2, focusing on role-playing.
🎯 Precise: Tuned with a low learning rate and cosine scheduler for 1 epoch to avoid overfitting while maximizing generalization.
📊 Training Details
Hardware: Single NVIDIA RTX 4090 (24GB)
Training Time: ~30 hours
Base Model: p-e-w/Qwen3-4B-Instruct-2507-heretic
Method: DoRA (Targeting all linear layers: q, k, v, o, gate, up, down)
Rank: 64 / Alpha: 128
Context Length: 4096 tokens
Precision: bfloat16 (with paged_adamw_8bit optimizer)
💻 Usage
Transformers
code
Python
download
content_copy
expand_less
```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "your-username/Qwen3-Heretic-4B-GrandMaster-DoRA"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
messages = [
{"role": "system", "content": "Ты полезный ассистен. Заточеный напомощь в ответе на вопросы.."},
{"role": "user", "content": "Write a Python function to solve the Knapsack problem using dynamic programming."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
GGUF (llama.cpp)
This model follows the licensing of the base Qwen model and the GrandMaster dataset. Please refer to the original repositories for detailed license information.
Trained with ❤️ using TRL and PEFT.