MannyLM/Synapse-V1

Synapse V1

Synapse V1 is a small decoder-only Transformer language model implemented from scratch in PyTorch (no torch.nn.Transformer / torch.nn.MultiheadAttention) and trained on the Tiny Shakespeare corpus at the character level.

• Parameters: ~826K (0.83M)
• Best validation loss: 1.6539 (character-level cross-entropy)
• Source code: https://github.com/GTX-Manish/Synapse-V1

⚠️ This is NOT a transformers model. It is a plain torch.nn.Module, so it cannot be loaded with AutoModel.from_pretrained. Load it with the small snippet under How to use, which uses the model.py included in this repo.

Model description

A standard GPT-style decoder-only Transformer with learned token and positional embeddings and a Pre-LN residual structure. Every component is hand-written:

• Multi-head causal self-attention — separate Q/K/V nn.Linear projections, manual head reshaping, scaled dot-product (Q·Kᵀ)/√d_head, a torch.tril causal mask applied via masked_fill(~mask, -inf) before softmax, then an output projection.
• Positional information — a learned nn.Embedding(max_sequence_length, d_model) added to the token embeddings.
• Pre-LN Transformer blocks — x = x + MHA(LayerNorm(x)) then x = x + FFN(LayerNorm(x)).
• Feed-forward network — Linear(d_model→d_ff) → ReLU → Linear(d_ff→d_model).
• LM head — final LayerNorm then Linear(d_model→vocab_size); embeddings are not tied.

Architecture / hyperparameters

Field	Value
Architecture	Decoder-only Transformer (from scratch)
Vocabulary size	65 (character-level)
Sequence length	128
d_model	128
num_heads	4
d_ff	512
num_layers	4
Activation	ReLU
Normalization	Pre-LN
Positional encoding	Learned
Parameters	~826K

Dataset

Tiny Shakespeare — a single ~1.1 MB text file of Shakespeare's works. The character-level tokenizer's vocabulary is the 65 unique characters in the training text. No other data is used.

Training procedure

• Objective: next-character prediction with token-level cross-entropy.
• Split: 80% train / 20% validation, by character index.
• Optimizer: AdamW (lr 1e-3, weight decay 0.01).
• Batch size: 32; sequence length: 128.
• Steps: 5000; validation measured every 500 steps over 50 batches.
• Selection: the checkpoint with the lowest validation loss is kept.
• Hardware: Apple Silicon (mps) / CPU.

Evaluation

Metric	Value
Best validation loss (character-level cross-entropy)	1.6539

No other metrics are reported. This is the only measured evaluation number for the model.

How to use

import json
import torch
from safetensors.torch import load_file
from huggingface_hub import hf_hub_download

# model.py is included in this repo (copied from the source project)
from model import Transformer

REPO = "MannyLM/Synapse-V1"

config = json.load(open(hf_hub_download(REPO, "config.json")))
tokenizer = json.load(open(hf_hub_download(REPO, "tokenizer.json")))
weights = load_file(hf_hub_download(REPO, "model.safetensors"))

model = Transformer(
    vocab_size=config["vocab_size"],
    max_sequence_length=config["max_sequence_length"],
    d_model=config["d_model"],
    num_heads=config["num_heads"],
    d_ff=config["d_ff"],
    num_layers=config["num_layers"],
)
model.load_state_dict(weights)
model.eval()

char_to_id = tokenizer["char_to_id"]
id_to_char = {int(k): v for k, v in tokenizer["id_to_char"].items()}


def generate(prompt, max_new_tokens=300, temperature=0.8, top_k=20):
    ids = [char_to_id[c] for c in prompt]
    for _ in range(max_new_tokens):
        ctx = torch.tensor([ids[-config["max_sequence_length"]:]])
        with torch.no_grad():
            logits = model(ctx)[:, -1, :] / temperature
        v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
        logits = torch.where(logits < v[:, -1:], torch.full_like(logits, -float("inf")), logits)
        probs = torch.softmax(logits, dim=-1)
        ids.append(torch.multinomial(probs, 1).item())
    return "".join(id_to_char[i] for i in ids)


print(generate("ROMEO:"))

Example output

Model output for the prompt ROMEO: (temperature=0.8, top_k=20), included verbatim:

ROMEO:
Why, sound the king; and that thou dost the colemater,
And who stands life, stay.

LUCIO:
O behold heavens, go we so have had been me thee,
Tonger what says that have it in the boate of me
While three well. Go, somethin

Intended use

Educational and research use: studying a correct, readable, from-scratch Transformer and its training pipeline, and generating Shakespeare-style character-level text for demos.

Limitations

• ~0.83M parameters trained on ~1 MB of a single author's text — it models only the style and characters of Tiny Shakespeare.
• Character-level with a 128-character context; no world knowledge, facts, or instruction following.
• Output is locally plausible but not globally coherent, and is not factual.
• Not a general-purpose assistant and not suitable for any downstream task requiring reliable, safe, or factual output.

Ethical considerations

The training data is public-domain Shakespeare. The model can only reproduce Shakespeare-style character sequences; it is not designed or safe for real-world decision-making, advice, or any user-facing application. Generated text is fiction-style and should not be treated as factual.

Reproducibility

All architecture hyperparameters and the tokenizer vocabulary are stored in config.json and tokenizer.json. Combined with model.py and model.safetensors, the exact model and tokenizer can be reconstructed. Full training code is in the GitHub repository.

Files

File	Description
config.json	Architecture hyperparameters.
model.safetensors	Model weights (~826K parameters, float32).
tokenizer.json	Character-level tokenizer (vocab + char_to_id / id_to_char).
model.py	Model definition (from scratch) used by the loader above.