akshit-mat/hermes-phase1-baseline

Hermes Phase 1 Baseline

Hermes Phase 1 Baseline is a small, experimental decoder-only multilingual Transformer language model trained from scratch. It supports English, Hindi (in Devanagari), and Hindi-English code-mixed text (Hinglish).

Model Architecture

• Type: Decoder-only Transformer
• Layers: 6
• Attention Heads: 6
• Embedding Dimension (d_model): 384
• Context Length: 320
• Vocabulary Size: 16,000
• Total Parameters: 16,913,664

Training Data (Phase 1 Corpus)

The model was trained on a custom Phase 1 dataset comprising three languages:

• English: 60,000,018 approximate tokens
• Hindi: 60,000,127 approximate tokens
• Hinglish: 60,000,021 approximate tokens
• Total: 180,000,166 approximate tokens

Data Splits

• Train: 161,962,678 tokens (3,032,462 examples)
• Validation: 8,945,810 tokens (168,142 examples)
• Test: 9,091,678 tokens (168,881 examples)

Checkpoint Details

• Training Step: 10,000
• Validation Loss: ~4.75
• Test Loss: ~4.79

Intended Use

This model is intended strictly as a research/educational artifact and baseline generation model. It serves to establish proof-of-concept multilingual behavior at an extremely small scale.

It is NOT production ready.

Limitations

Due to its small parameter count (16.9M) and abbreviated training schedule (10,000 steps), the model exhibits substantial limitations:

• Repetitive Text: The model often devolves into repeating identical characters or tokens.
• Weak Coherence: It struggles to maintain semantic coherence beyond a few words.
• No Instruction Tuning: This is a base next-token prediction model. It cannot answer questions, engage in multi-turn chat, or follow instructions.
• Malformed Continuations: Expected on highly complex language features.

Example Generations (Greedy Decoding)

These examples were generated using the Day 9 Baseline Checkpoint with greedy decoding (temperature=1.0, sampling disabled).

English

• Prompt: "The rapid development of artificial intelligence"
• Generated: –18" and the P"

Hindi

• Prompt: "भारत का इतिहास बहुत पुराना और"
• Generated: प्याहाहाहाहाहाहाहाहाहाहाहाहाहा

Hinglish

• Prompt: "Mujhe lagta hai ki yeh movie"
• Generated: hai