Workstation5495/Resonatex-D3

ResonateX D3

ResonateX D3 is a small GPT-2 architecture causal language model (approximately 120–130M parameters, 12 layers, 12 attention heads, 768 hidden size) trained from scratch as a conversational assistant. It is the successor to ResonateX D2, keeping D2's dedicated <|user|> / <|assistant|> / <|endofturn|> turn format but with a larger architecture and a much longer 1024-token context window. It is designed to run fully offline on consumer hardware (CPU, or GPU/MPS if available).

Model Details

• Architecture: GPT-2 (gpt2 model type, GPT2LMHeadModel)
• Trained from scratch: yes — no pretrained weights were used as a starting point
• Parameters: ~120–130M
• Layers (n_layer): 12
• Attention heads (n_head): 12
• Attention head size: 64
• Hidden size (n_embd): 768
• Feed-forward size (n_inner): 3072
• Activation: gelu_new
• Dropout: residual 0.05, embedding 0.05, attention 0.05
• Context length: 1024 tokens
• Vocabulary size: 50,260 (base ai-forever/rugpt3small_based_on_gpt2 vocabulary plus 3 added special tokens — confirm against the actual tokenizer_config.json before publishing, in case the D3 tokenizer differs from D2's)
• Language(s): Russian and English dialogue data
• Base tokenizer: ai-forever/rugpt3small_based_on_gpt2 (model weights were trained from scratch, not fine-tuned from this checkpoint — only its tokenizer was reused)
• Causal language modeling: yes
• Assistant-only loss masking: yes
• Multi-turn dialogue: yes
• License: update this field with the actual license that applies to your weights and training data before publishing (note that some source datasets may carry their own license terms that also need to be respected)

Chat / Prompt Format

Unlike a plain text-completion model, ResonateX D3 was trained on turns wrapped in explicit role tokens — the same format used by D2:

<|user|>
{user message}
<|endofturn|>
<|assistant|>
{assistant reply}
<|endofturn|>

Training loss was computed only on assistant turns (user turns are masked out of the loss), so the model is specifically optimized to produce assistant replies, not to continue arbitrary text. To generate a reply, build a prompt ending right after <|assistant|>\n and let the model continue from there; generation should stop at the next <|endofturn|> (or if the model drifts, at the next <|user|> token).

Intended Uses

This model is intended for:

• Experimentation with small, locally-run conversational language models.
• Educational and research use around GPT-2 style architectures and simple chat-formatted pretraining.
• Lightweight offline chat applications where a large hosted LLM is not available, required, or desired.

It is not intended for:

• Production use requiring factual accuracy, safety guarantees, or content moderation, none of which this model provides on its own.
• High-stakes decision-making of any kind (medical, legal, financial, etc.).
• Reliable arithmetic or logical reasoning beyond very simple synthetic examples, if such examples were included in training data (see Training Data below).

Limitations and Bias

• As a small (~120–130M parameter) model, ResonateX D3 has limited world knowledge and reasoning capability compared to larger language models. Responses may be short, generic, repetitive, or factually incorrect.
• The model has a fixed context window of 1024 tokens; long conversations will have earlier turns truncated.
• Training data composition, and any biases, factual errors, or inappropriate content it may contain, should be documented in the Training Data section below before publishing. No dedicated bias or safety evaluation has been performed.
• The model has no built-in safety filtering. Applications built on top of it should implement their own moderation if needed.
• Because generation stops on custom special tokens, using this model through a generic pipeline without setting eos_token_id to include the <|endofturn|> token id may cause it to keep generating past the intended end of a reply.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "path/to/resonatex-d3"  # local path or Hugging Face repo id

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
model.eval()

eot_id = tokenizer.convert_tokens_to_ids("<|endofturn|>")

prompt = "<|user|>\nПривет! Кто ты?\n<|endofturn|>\n<|assistant|>\n"

inputs = tokenizer(prompt, return_tensors="pt")

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=192,
        do_sample=True,
        temperature=0.7,
        top_p=0.92,
        top_k=50,
        repetition_penalty=1.12,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
        eos_token_id=[tokenizer.eos_token_id, eot_id],
    )

response = tokenizer.decode(
    output_ids[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=False,
)

# Trim anything the model generated past its own turn.
response = response.split("<|endofturn|>")[0].split("<|user|>")[0].strip()

print(response)

Prompting in a different format (e.g. D1's plain "Вопрос: ... Ответ:" style) is likely to reduce output quality significantly, since this is not the format D3 was trained on. max_new_tokens can now be pushed considerably higher than on D2 (up to ~900) thanks to the larger 1024-token context, as long as enough room is left for the prompt and history.

Local Demo Application

A local Gradio-based chat interface for this model, called ResonateX D3, is available separately. It loads the model directly from disk, requires no internet connection at runtime, builds prompts using the <|user|> / <|assistant|> / <|endofturn|> format above, and exposes temperature, max output tokens (up to 896), top-p, and repetition penalty as adjustable generation settings in the UI. It also keeps more turns of chat history in the prompt than the D2 interface did, to take advantage of the larger context window.

Requirements: Python 3.10+, gradio, torch, transformers.

Training Data

Not filled in — the specific datasets, row counts, and preprocessing used for D3 were not provided. Fill in this section (sources, rows used, filtering/deduplication steps) before publishing this model card, following the same structure as D2's Training Data table.

Training Procedure

• Initialization: random weights (trained from scratch), using the tokenizer from ai-forever/rugpt3small_based_on_gpt2 extended with 3 special tokens (<|user|>, <|assistant|>, <|endofturn|>)
• Objective: causal language modeling with assistant-only loss masking (loss computed only on assistant turn tokens; user turn tokens and role tags are masked out with a -100 label)
• Sequence length: up to 1024 tokens per example
• Optimizer: AdamW (adamw_torch)
• Learning rate: 1.5e-4, cosine schedule, 5% warmup ratio
• Weight decay: 0.01
• Batch size: 2 per device, gradient accumulation 8 steps (effective batch size 16)
• Precision: fp16 (TF32 disabled)
• Gradient checkpointing: enabled
• Group by length: enabled
• GPU: NVIDIA T4
• Step budget: up to 8,000 steps, additionally capped by a wall-clock time limit (175 minutes), whichever came first
• Checkpointing: every 500 steps
• Train/validation split: 98% / 2%, seeded split
• Seed: 42 (for Python random, dataset shuffling/splitting, and transformers.set_seed)

Evaluation

Not filled in here. If you have metrics.json / final trainer.evaluate() output from your training run (loss, perplexity, steps completed, training time), add it to this section before publishing this model card publicly.

Citation

If you use this model, please cite it as:

@misc{resonatex-d3,
  title  = {ResonateX D3},
  author = {<add author/organization name>},
  year   = {<add year>},
}