Workstation5495/Resonatex-D2-72M-Experimental

ResonateX D2

ResonateX D2 is a small GPT-2 architecture causal language model (approximately 72M parameters, 8 layers, 10 attention heads, 640 hidden size) trained from scratch as a conversational assistant. It is the dialogue-focused successor to ResonateX D1, using a dedicated <|user|> / <|assistant|> / <|endofturn|> turn format instead of D1's plain question/answer text template. It is designed to run fully offline on consumer hardware (CPU, or GPU/MPS if available).

Model Details

• Architecture: GPT-2 (gpt2 model type, GPT2LMHeadModel)
• Parameters: ~71,883,520
• Layers: 8
• Attention heads: 10
• Hidden size: 640
• Feed-forward size (n_inner): 2560
• Vocabulary size: 50,260 (base ai-forever/rugpt3small_based_on_gpt2 vocabulary plus 3 added special tokens)
• Context length: 512 tokens
• Language(s): Russian and English dialogue data.
• Base tokenizer: ai-forever/rugpt3small_based_on_gpt2 (model weights were trained from scratch, not fine-tuned from this checkpoint - only its tokenizer was reused)
• License: update this field with the actual license that applies to your weights and training data before publishing (note that some source datasets below carry their own license terms that also need to be respected).

Chat / Prompt Format

Unlike a plain text-completion model, ResonateX D2 was trained on turns wrapped in explicit role tokens:

<|user|>
{user message}
<|endofturn|>
<|assistant|>
{assistant reply}
<|endofturn|>

Training loss was computed only on assistant turns (user turns are masked out of the loss), so the model is specifically optimized to produce assistant replies, not to continue arbitrary text. To generate a reply, build a prompt ending right after <|assistant|>\n and let the model continue from there; generation should stop at the next <|endofturn|> (or if the model drifts, at the next <|user|> token).

Intended Uses

This model is intended for:

• Experimentation with small, locally-run conversational language models.
• Educational and research use around GPT-2 style architectures and simple chat-formatted fine-tuning/pretraining.
• Lightweight offline chat applications where a large hosted LLM is not available, required, or desired.

It is not intended for:

• Production use requiring factual accuracy, safety guarantees, or content moderation, none of which this model provides on its own.
• High-stakes decision-making of any kind (medical, legal, financial, etc.).
• Reliable arithmetic or logical reasoning beyond the very simple synthetic examples described below.

Limitations and Bias

• As a small (~72M parameter) model, ResonateX D2 has limited world knowledge and reasoning capability compared to larger language models. Responses may be short, generic, repetitive, or factually incorrect.
• The model has a fixed context window of 512 tokens; long conversations will have earlier turns truncated.
• The synthetic math and logic examples used in training cover only basic arithmetic (addition, subtraction, multiplication of numbers up to 999) and simple numeric comparisons - the model should not be trusted for anything beyond that.
• Training data was drawn from crowd-sourced and community dialogue datasets (see below); any biases, factual errors, or inappropriate content present in those sources may be reflected in the model's outputs. No dedicated bias or safety evaluation has been performed.
• The model has no built-in safety filtering. Applications built on top of it should implement their own moderation if needed.
• Because generation stops on custom special tokens, using this model through a generic pipeline without setting eos_token_id to include the <|endofturn|> token id may cause it to keep generating past the intended end of a reply.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "path/to/resonatex-d2"  # local path or Hugging Face repo id

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
model.eval()

eot_id = tokenizer.convert_tokens_to_ids("<|endofturn|>")

prompt = "<|user|>\nПривет! Кто ты?\n<|endofturn|>\n<|assistant|>\n"

inputs = tokenizer(prompt, return_tensors="pt")

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.7,
        top_p=0.92,
        top_k=50,
        repetition_penalty=1.12,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
        eos_token_id=[tokenizer.eos_token_id, eot_id],
    )

response = tokenizer.decode(
    output_ids[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=False,
)

# Trim anything the model generated past its own turn.
response = response.split("<|endofturn|>")[0].split("<|user|>")[0].strip()

print(response)

Prompting in a different format (e.g. D1's plain "Вопрос: ... Ответ:" style) is likely to reduce output quality significantly, since this is not the format D2 was trained on.

Local Demo Application

A local Gradio-based chat interface for this model, called ResonateX D2, is available separately. It loads the model directly from disk, requires no internet connection at runtime, builds prompts using the <|user|> / <|assistant|> / <|endofturn|> format above, and exposes temperature, max output tokens, top-p, and repetition penalty as adjustable generation settings in the UI.

Requirements: Python 3.10+, gradio, torch, transformers.

Training Data

The model was trained on a mix of dialogue and small synthetic datasets:

Source	Description	Rows used (limit)
IlyaGusev/saiga_scored	Russian instruction/chat dialogues	up to 12,000
OpenAssistant/oasst1	Community-sourced assistant conversations, filtered to Russian and English, reconstructed into full conversation branches	up to 18,000
Synthetic arithmetic	Generated addition/subtraction/multiplication questions with numbers up to 999	1,000
Synthetic comparison/logic	Generated "which number is bigger" style questions	1,000
Synthetic multi-turn context	Small hand-written templates testing short-term memory of stated facts (name, interests, etc.) across a conversation	1,000
Synthetic identity examples	Hand-written "who are you / what is your name" turns identifying the model as ResonateX AI D2	7 unique examples, repeated 20x

All dialogue examples were cleaned (control character and whitespace normalization, length filtering, garbage/URL-spam filtering) and deduplicated by content hash, both within each source and again across the combined dataset, before tokenization.

Training Procedure

• Initialization: random weights (trained from scratch), using the tokenizer from ai-forever/rugpt3small_based_on_gpt2 extended with 3 special tokens (<|user|>, <|assistant|>, <|endofturn|>).
• Objective: causal language modeling with assistant-only loss masking (loss computed only on assistant turn tokens; user turn tokens and role tags are masked out with a -100 label).
• Sequence length: up to 512 tokens per example.
• Optimizer: AdamW (adamw_torch)
• Learning rate: 2e-4, cosine schedule, 5% warmup ratio
• Weight decay: 0.01
• Batch size: 8 per device, gradient accumulation 4 steps (effective batch size 32)
• Precision: fp16 on CUDA, fp32 on CPU
• Max gradient norm: 1.0
• Step budget: up to 7,000 steps, additionally capped by a wall-clock time limit (100 minutes) via a custom training callback, whichever came first
• Train/validation split: 98% / 2%, seeded split
• Seed: 42 (for Python random, dataset shuffling/splitting, and transformers.set_seed)

Evaluation

Not filled in here. If you have metrics.json / final trainer.evaluate() output from your training run (loss, perplexity, steps completed, training time), add it to this section before publishing this model card publicly.

Citation

If you use this model, please cite it as:

@misc{resonatex-d2,
  title  = {ResonateX D2},
  author = {<add author/organization name>},
  year   = {<add year>},
}