TinyTalk-124M

TinyTalk-124M is a 124-million-parameter conversational language model built entirely from scratch in PyTorch — custom transformer architecture, custom training loop, and a two-stage training pipeline (base pretraining followed by LoRA instruction fine-tuning).

Model Details

• Architecture: Custom decoder-only transformer
  • 12 transformer blocks
  • Embedding dimension: 512
  • 8 attention heads
  • Rotary Position Embeddings (RoPE)
  • SwiGLU feed-forward layers
  • Causal self-attention via scaled dot-product attention
• Vocabulary: GPT-2 tokenizer (50,257 tokens)
• Context length: 512 tokens
• Total parameters: ~124.2M
• Format: safetensors (fp32)

Training Pipeline

Stage A — Base Pretraining The model was pretrained from scratch on general text data to learn language modeling fundamentals.

Stage B — Conversational Fine-Tuning (LoRA) The pretrained base was fine-tuned using LoRA (Low-Rank Adaptation) on a ChatAlpaca-style instruction/response dataset (~68K examples) to teach instruction-following and conversational behavior. LoRA was applied to attention (qkv_proj, out_proj) and feed-forward (w1, w2, w3) projections, then merged back into the base weights for standalone deployment.

• LoRA rank: 64, alpha: 128
• Effective batch size: 32 (batch size 16 × grad accumulation 2)
• Mixed precision (bf16) training
• Cosine learning rate schedule with warmup
• Early stopping based on validation loss

Final validation perplexity: ~25.8

Prompt Format

The model expects an Alpaca-style instruction format:

### Instruction:
{your instruction here}

### Response:

Intended Use

This model is a small-scale educational/portfolio project demonstrating an end-to-end from-scratch LLM pipeline: architecture design, pretraining, parameter-efficient fine-tuning, and deployment. It performs reasonably on everyday conversational instructions but is not intended for production use, factual reliability, or specialized/technical domains.

Limitations

• Scale: At 124M parameters, this model has limited world knowledge and reasoning capability compared to larger LLMs.
• Domain coverage: Performs noticeably better on general conversational prompts (matching its fine-tuning data) than on specialized technical or factual topics, where output coherence can degrade.
• Not factually reliable: Outputs should not be trusted for factual accuracy, especially on scientific, medical, legal, or technical subjects.
• English only.

Example

Input:

### Instruction:
Write a short poem about the ocean.

### Response:

Output:

The sea is filled with stars, rapping deep in the sky and its rays float so high
that it can see as far away as the horizon. It has been through for centuries,
since we have lived to this day.

Citation / Acknowledgment

Built as a personal/educational project exploring transformer architecture design, staged LLM training (pretraining + LoRA fine-tuning), and end-to-end model deployment. ```