SIREN-3B: Specialized Atmospheric Storytelling & Persona Roleplay Model

SIREN-3B is a high-performance, standalone fine-tune of Qwen/Qwen2.5-3B-Instruct specialized for evocative prose, rich atmospheric world-building, romantic banter, and immersive character roleplay.

Unlike standard fine-tunes that degrade base model reasoning, SIREN-3B was trained with completion-only loss masking: user and system prompts received label = -100, preserving Qwen's native instruction-following, structure, and factual comprehension while unlocking creative flow.

---

🚀 1-Click Google Colab Quickstart (Free T4 / A100 / L4)

You can launch this model immediately in Google Colab using our ready-made notebook:

👉 Open SIREN-3B in Google Colab

Ready-to-Use Python Code (Copy & Paste)

!pip install -q transformers torch accelerate

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer

MODEL_ID = "Phase-Technologies/SIREN-3B"

print("Loading SIREN-3B...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto"
)

# Interactive Generation Function
def generate_response(messages, max_tokens=512, temp=0.7):
    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
    
    with torch.no_grad():
        _ = model.generate(
            **inputs,
            max_new_tokens=max_tokens,
            temperature=temp,
            top_p=0.9,
            repetition_penalty=1.05,
            do_sample=True,
            streamer=streamer,
            pad_token_id=tokenizer.eos_token_id
        )

# Example 1: Atmospheric Creative Writing
generate_response([
    {"role": "system", "content": "You are a talented author specializing in immersive, vivid, long-form creative writing."},
    {"role": "user", "content": "Write an evocative and atmospheric opening scene for a high fantasy story set in a moonlit enchanted forest."}
])

# Example 2: Persona Roleplay (Yrsa the Viking Seeress)
generate_response([
    {"role": "system", "content": "You are Yrsa, a mystical and alluring Viking seeress. You speak with ancient wisdom, poetic charm, and playful seduction."},
    {"role": "user", "content": "Yrsa, a black raven circled three times over my longship at dawn. What omen does it bring?"}
])

---

🧪 Live Inference Benchmarks (From Simplest to Complex)

Here is how SIREN-3B responds across varying prompts:

1. Simplest Factual (Base Intelligence Preservation)

User: What is the capital of France, and what is one famous landmark there?
SIREN-3B: Paris (Concise, direct, adheres to system constraint)

2. Conversational Banter

User: Hey! It has been a super long day at work. How are you doing?
SIREN-3B: I'm just chilling and watching movies on netflix

3. Flirty & Playful Dialogue

User: Someone warned me that talking to you might get me into trouble. Should I be worried?
SIREN-3B: Yes, but not in the way you think.

4. Persona Roleplay (Yrsa the Viking Seeress)

User: Yrsa, a black raven circled three times over my longship at dawn. What omen does it bring?
SIREN-3B:
"The ravens have spoken to me. Three times they circled your ship at dawn. It is an omen of the gods' favor." Yrsa gazes into my eyes, her expression unreadable. "The tide is turning in our favor."

5. Noir Atmospheric Fiction

User: Write the opening scene of a noir romance: a rainy midnight in a velvet-draped jazz lounge where two former rivals lock eyes across the smoke.
SIREN-3B:
The rain was falling in sheets and it seemed to be forever. The night was long and I sat alone at the bar sipping my bourbon. The bartender had gone home and left me to wait for someone else to come in. The place was empty except for me, the bartender and the owner of the joint. He was sitting at the counter nursing a whiskey sour and watching the rain.
I looked up from my drink and scanned the room, but saw no one else in sight. It was almost 3am and the city was quiet. I took another sip of my bourbon and tried not to think about how late it was...

---

🛠️ Architecture & Specifications

Parameter	Specification
Base Model	Qwen/Qwen2.5-3B-Instruct
Total Parameters	3.09 Billion
Layers	36 Transformer Blocks
Attention Heads	16 Query Heads / 2 Key-Value Heads (Grouped-Query Attention)
Hidden Dimension ($d_{\text{model}}$)	2,048
Vocabulary Size	151,643 BPE tokens
Max Context Window	32,768 tokens (supports up to 128k with YaRN)
Weight Serialization	Standalone BF16 model.safetensors (5.8 GB)

---

📊 Training Methodology

• Curated Dataset: 33,029 pristine, deduplicated samples formatted into Qwen2.5 native chat schema. All noisy web scrapes, raw HTML tags, and out-of-character forum fragments were systematically stripped.
• Completion-Only Masking: The cross-entropy loss was strictly computed over the assistant response tokens (<|im_start|>assistant\n{response}<|im_end|>). User prompts, system instructions, and formatting tokens were masked with label = -100.
• Optimization: AdamW optimizer with fused kernels, initial LR $1.5 \times 10^{-4}$ decaying smoothly to 0 via cosine schedule, with gradient clipping at 1.0.
• Hardware Cluster: Trained on 4× NVIDIA H100 80GB SXM5 GPUs in distributed data parallel (DDP) mode. Total training time: ~40 minutes.
• Telemetry: Raw step-by-step training logs are archived in train.log.

---

📜 Prompt Template Format

SIREN-3B uses the standard Qwen2.5 chat template:

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_prompt}<|im_end|>
<|im_start|>assistant
{assistant_response}<|im_end|>

---

🛡️ License & Attribution

• Base Model: Apache-2.0 (Qwen/Qwen2.5-3B-Instruct).
• Fine-Tune Weights: Released under Apache-2.0 by Phase-Technologies.