YuRiVeRTi/TEROM-DUST-1

TEROM DUST 1

An experimental English-literature-focused language model by TESROL.

Model details

• Parameters: 30,259,712
• Architecture: Custom decoder-only Transformer
• Layers: 7
• Hidden dimension: 512
• Attention heads: 8
• Vocabulary: 16,000
• Maximum context length: 1,024 tokens
• Position encoding: RoPE
• Activation: GELU
• Input/output embeddings: Tied
• Tokenizer: Custom byte-level BPE

Training

This checkpoint was trained from scratch for 10 epochs, with 1,540 optimizer updates.

Best validation loss: 5.20611759 Best validation perplexity: 182.38459013 Best epoch: 9

The corpus was Shakespeare-heavy. Shakespeare accounts for approximately 83.58% of training tokens.

Limitations

This is an experimental research checkpoint, not a production chatbot. Evaluations found fragmented outputs, weak narrative continuity, limited prompt sensitivity, and a strong tendency toward Shakespeare-like dialogue.

It should not be described as a reliable creative-writing model.

Dataset and licensing

The original training corpus is not included in this repository. Source and edition rights have not been cleared for every jurisdiction. This initial release does not grant a separate license for reuse or redistribution of the weights.

Loading

This repository contains a custom PyTorch architecture. It is not directly loadable through the standard AutoModelForCausalLM interface.

See modeling_terom.py for the architecture definition.