SnifferCaptain/YModel3.2-fm-1B

this file is generated by Qwen3.8-27B under human supervision

模型描述

YModel3.2-fm-1B 是 YModel3.2 家族的 1B 参数基础权重模型（foundation weights）。

模型细节

项目	值
架构	YModel3.2
Scale level	1
参数量	1,120,343,553（1.12B，含嵌入）
Hidden size	1024
块数	31（6 个 FullBlock + 25 个 GroupBlock）
layer pattern	fgggggfgggggfgggggfgggggfgggggf
注意力头 / KV 头	8 / 2（GQA）
头维度	256
Group size	8（固定分组）
FFN	SwiGLU + SEBlock，intermediate 3072
DCP	dynamic compression probe（probe hidden 1024）；use_dcp=false，分组固定，probe 权重保留
Sengram	bucket 16384 / topk 12
词表	151,669（Qwen3 tokenizer，tied embeddings）
上下文	8192
精度	bfloat16

Group Attn 与 Token Superposition

YModel3.2-fm 的 group attn 主要借鉴了论文 Efficient Pre-Training with Token Superposition（Bowen Peng, Théo Gigant, Jeffrey Quesnelle, arXiv:2605.06546）的思想：将连续 token 分组（superpose）为一个整体进行高效处理。论文提出的 Token-Superposition Training（TST）把多个连续 token 合并为一个 bag、以 multi-hot 交叉熵（MCE）目标训练，再经 recovery 阶段回到标准训练，在 10B A1B 规模上报告了至多 2.5x 的预训练时间缩减。本模型的 group attn 是其架构侧的延伸：GroupBlock 以 group_size=8 的分组处理 token，FullBlock / GroupBlock 按 f/g pattern 交错排布，并用 F2G / G2F 层完成组与完整 token 流之间的信息转换。

训练细节

训练分四个阶段完成（bf16，AdamW 与 Muon 双优化器）：

1. 预训练：causal LM，1024 上下文，8.52B Tokens，冻结嵌入与 lm_head。
2. SFT：UltraData SFT + Instant 混合语料，8192 上下文（预训练为 1024，此阶段首次扩展到 8192），Muon 5e-6 / AdamW 5e-7，嵌入与 lm_head 冻结，4.14B Tokens。
3. 蒸馏：teacher 为 Qwen3-0.6B（CE=0.5 / KL=0.5，temperature=1.0），固定 group_size=4，DCP probe 冻结，哈希辅助监督加强至 1e-1，Muon 1e-4 / AdamW 1e-5，0.98B Tokens。

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "SnifferCaptain/YModel3.2-fm-1B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype="auto",
)

对话格式

模型使用 ChatML 风格的对话边界。完整的输入可以写成：

<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{user content}<|im_end|>
<|im_start|>assistant
<think>{thinking_definitions}
{reasoning content}</think>

{assistant content}<|im_end|>

其中，{thinking_definitions} 是紧跟在 <think> 后面的第一行控制信息，不是思考正文。它可以是：

• juice = 1.23：思考预算提示，可以使用小数。
• structured = true：表示结构化思考。
• juice = 1.23; structured = true：同时指定预算和结构化思考。

控制行结束后换行，再写入 {reasoning content}；思考结束使用 </think>，之后空一行写最终的 {assistant content}。自由形式思考可以省略 structured，也可以显式写成 structured = false。如果没有控制字段，保留空的控制行即可：

<think>
{reasoning content}
</think>

每个角色都使用 <|im_start|>role\n...<|im_end|> 边界；system 和 user 内容不会放入 <think>，思考与最终回答只属于 assistant 消息。

注意：该架构的 generate() 为模型自定义实现，不使用 KV cache（每步重算完整前缀），长序列生成会相应变慢。

Notes

• 包含自定义代码：modeling_ymodel32.py（自包含：配置类 + 完整架构）、ymodel32_eval.py（参考实现），加载需 trust_remote_code=True。
• 权重为单个 model.safetensors（bfloat16，约 2.55GB）。
• 基准分数与示例输出待补充。

---

Model Description

YModel3.2-fm-1B is the 1B-parameter foundation-weights model of the YModel3.2 family.

Model Details

Item	Value
Architecture	YModel3.2
Scale level	1
Parameters	1,120,343,553 (1.12B, incl. embeddings)
Hidden size	1024
Blocks	31 (6 FullBlocks + 25 GroupBlocks)
Layer pattern	fgggggfgggggfgggggfgggggfgggggf
Attention heads / KV heads	8 / 2 (GQA)
Head dim	256
Group size	8 (fixed grouping)
FFN	SwiGLU + SEBlock, intermediate 3072
DCP	dynamic compression probe (probe hidden 1024); use_dcp=false, fixed grouping, probe weights retained
Sengram	bucket 16384 / topk 12
Vocab	151,669 (Qwen3 tokenizer, tied embeddings)
Context	8192
Dtype	bfloat16

Group Attn and Token Superposition

The group attn of YModel3.2-fm is inspired primarily by Efficient Pre-Training with Token Superposition (Bowen Peng, Théo Gigant, Jeffrey Quesnelle, arXiv:2605.06546): superposing contiguous tokens into a group and processing them as a single unit. The paper's Token-Superposition Training (TST) merges many contiguous tokens into one bag and trains with a multi-hot cross-entropy (MCE) objective, then recovers to standard training, reporting up to a 2.5x pre-training time reduction at 10B A1B scale. This model's group attn is the architectural extension of that idea: GroupBlocks process tokens in groups of 8, FullBlocks and GroupBlocks are interleaved per the f/g pattern, and F2G / G2F layers transfer information between the group and full-token streams.

Training Details

Training ran in four stages (bf16, AdamW + Muon dual optimizers):

1. Pre-training: causal LM, 1024 context, 8.52B Tokens, embeddings and lm_head frozen.
2. SFT: mixed UltraData SFT + Instant corpus, 8192 context (pre-training used 1024, so 8192 is new context here), Muon 5e-6 / AdamW 5e-7, embeddings and lm_head frozen, 4.14B Tokens.
3. Distillation: teacher Qwen3-0.6B (CE=0.5 / KL=0.5, temperature 1.0), fixed group_size=4, DCP probe frozen, hash auxiliary supervision raised to 1e-1, Muon 1e-4 / AdamW 1e-5, 0.98B Tokens.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "SnifferCaptain/YModel3.2-fm-1B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype="auto",
)

Chat Format

The model uses ChatML-style turn boundaries. A complete input can be written as:

<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{user content}<|im_end|>
<|im_start|>assistant
<think>{thinking_definitions}
{reasoning content}</think>

{assistant content}<|im_end|>

{thinking_definitions} is the first control line immediately after <think>, not the reasoning body. It may be:

• juice = 1.23 for a numeric reasoning-budget hint;
• structured = true to mark structured reasoning; or
• juice = 1.23; structured = true to provide both fields.

Terminate the control line with a newline, then place the reasoning in {reasoning content}. Close it with </think>, leave one blank line, and put the final answer in {assistant content}. For free-form reasoning, structured may be omitted or written as structured = false. If no control field is needed, keep an empty control line:

<think>
{reasoning content}
</think>

Note: this architecture ships a custom generate() implementation without KV caching (the full prefix is recomputed each step), so long-sequence generation is correspondingly slower.

Notes

• Custom code included: modeling_ymodel32.py (self-contained: config class + full architecture) and ymodel32_eval.py (reference implementation); load with trust_remote_code=True.
• Weights: single model.safetensors (bfloat16, ~2.55GB).
• Benchmark scores and sample outputs: to be added.