DarkWhiteProductions/urvashi-gemma3-4B-Base

URVASHI-Gemma3

URVASHI-Gemma3 is a Gemma-3 compatible decoder-only transformer featuring factorized embeddings initialized through SVD compression of Gemma token embeddings.

Features

• Gemma-3 compatible geometry
• Factorized embeddings
• Grouped Query Attention (GQA)
• Sliding Window Attention (SWA)
• RoPE positional embeddings
• RMSNorm
• HuggingFace compatible

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "shashwatmudgal/urvashi-gemma3-4B-Base",
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(
    "shashwatmudgal/urvashi-gemma3-4B-Base"
)

prompt = "The ancient city stood silent beneath"

inputs = tokenizer(
    prompt,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
)

print(
    tokenizer.decode(outputs[0])
)

Notes

This repository requires:

trust_remote_code=True

because it uses custom URVASHI architecture files.

Citation

@misc{urvashi2026,
  title={URVASHI},
  author={Divyansh Bharadwaj},
  year={2026}
}