YNSScarSaiyan/vegeta-weights

Vegeta public weights

Public weight dumps for Vegeta SFT. This is not YNSScarSaiyan/vegeta (seed / older files). Do not treat that repo as this one.

Vegeta is a looped decoder: 2 shared transformer blocks run for 16 iterations, with per-iteration LayerNorms (architecture v2). Not a Hugging Face AutoModel.

Created by Dakuwon Moody.

Status

Item	Value
Public role	Weights only
Class	LoopedLLM in vegeta_model_v2.py
Parameters	~0.73B (untied embed + LM head, 2 shared layers, 16-iter norms)
Tokenizer	Qwen/Qwen1.5-0.5B-Chat (vocab 151,936)
Training	Packed next-token CE on the same agent HF mix as SImi / Uni
Hardware	AMD Instinct MI300X leftover HBM (mem fraction 0.10), bf16, batch 1, seq 1024
LR	5e-6
Save cadence	every 500 steps under sft/step_<n>/
Latest public file	sft/step_67000/vegeta_sft_step_67000.pt
Hosted inference	Not supported

Checkpoint 500 is architecture v2 (iter_attn_norms / iter_ffn_norms). Keep that restore seed. Later sft/step_* files are the continuing SFT life.

Architecture

Property	Value
Hidden size	2,048
Shared layers	2
Loop iterations	16
Attention heads	32
Head dim	64
MLP	2048 -> 8192 GELU -> 2048
Max position	1,024
Embeddings	untied embed + lm_head
Extra	pos_embed, iter_embed, per-iter attn/ffn LayerNorms

v1 (shared layernorms + iter_gate) is only for older keys. Live loads v2 when iter_attn_norms is in the state dict.

Checkpoints

Path	What
Seed vegeta_sft_step_500.pt	v2 restore base (lives on the seed repo, not required in this tree)
sft/step_<n>/vegeta_sft_step_<n>.pt	Agent-loop SFT dumps (model_state_dict, step, loss)

Local copies are deleted only after the .pt exists here.

Load

import torch
from vegeta_model_v2 import LoopedLLM, LoopedConfig

cfg = LoopedConfig(
    vocab_size=151936,
    hidden_size=2048,
    num_attention_heads=32,
    num_hidden_layers=2,
    max_position_embeddings=1024,
    loop_iterations=16,
)
model = LoopedLLM(cfg)
blob = torch.load("vegeta_sft_step_67000.pt", map_location="cpu", weights_only=False)
state = blob.get("model_state_dict") or blob.get("model") or blob
model.load_state_dict(state, strict=False)
model = model.to(dtype=torch.bfloat16)
model.eval()

Tokenizer: Qwen/Qwen1.5-0.5B-Chat.

Training mix

Same full HF streams as SImi and Uni:

• instruction: Tulu-3, OpenHermes-2.5, UltraChat
• agent / tool: Orca AgentInstruct, Agent-FLAN, xLAM, ToolACE, Glaive, Hermes function calling

Packed LM (not prompt-masked SFT). Expect step loss in the mid-single digits; that is the mix, not a wipe.

Sisters

• SImi (JAX 2.32B): YNSScarSaiyan/simi-weights
• Uni (planetary PyTorch): YNSScarSaiyan/uni

Not this

• Not YNSScarSaiyan/vegeta made public
• Not Transformers-native
• Not a Qwen fine-tune of the 0.5B-Chat weights (tokenizer only)