Lara-350M (Pretrained)

A 350M parameter LLaMA-style language model pretrained from scratch on a custom corpus. This is the base pretrained checkpoint — it has not yet been instruction-tuned or fine-tuned with any persona.

Architecture

• Type: LLaMA (decoder-only transformer)
• Parameters: 439,403,520
• Layers: 24
• Hidden size: 1024
• Attention heads: 16
• KV heads: 16
• Vocab size: 64,000
• Max sequence length: 2048
• Activation: SiLU (SwiGLU)
• Normalization: RMSNorm

Tokenizer

Important: This model uses a custom tokenizer trained from scratch on the training corpus. It is NOT a standard LLaMA, GPT-2, or any other pre-existing tokenizer.

• Type: BPE (Byte-Pair Encoding)
• Vocab size: 64,000
• Special tokens: BOS (<s>), EOS (</s>), UNK (<unk>)
• Training: Trained alongside the model on the same corpus

Do NOT use a standard LLaMA tokenizer with this model — the token IDs will be completely wrong and produce garbled output.

Training

• Architecture: Custom LLaMA-350M (trained from scratch)
• Training: 100,000 iterations with AdamW optimizer
• Batch size: 32 effective (2 micro-batch x 8 grad accum x 2 GPUs)
• Learning rate: 3e-4
• Hardware: 2x NVIDIA RTX 3060 (12GB VRAM each)
• Corpus: Custom pretraining corpus

What This Is Not

This is a raw pretrained language model. It has:

• No instruction tuning
• No persona or personality
• No RLHF or DPO alignment
• No guardrails or safety training

It predicts the next token based on patterns learned during pretraining. To use it as an assistant, it would need further fine-tuning (SFT, RLHF, etc.).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("0ldev/Lara-350M")
tokenizer = AutoTokenizer.from_pretrained("0ldev/Lara-350M")

inputs = tokenizer("The meaning of life is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

Apache 2.0