Topic Classifier Demo
⚖
Classify a question by corresponding legal topic
Smoliakov PRO
Yehor
AI & ML interests
Speech-to-Text, Text-to-Speech, Voice over Internet Protocol
Recent Activity
published
a Space
1 day ago
Yehor/question-classifier-demo
updated
a Space
1 day ago
Yehor/question-classifier-demo
commented on
an
article
2 days ago
Welcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM
Organizations
Yehor's activity
published
a
Space
1 day ago
updated
a
Space
1 day ago
commented on
Welcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM
2 days ago
Nice!
upvoted
an
article
2 days ago
Article
Welcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM
•
237
liked
a
model
2 days ago
reacted to
eliebak's
post with 🔥
2 days ago
Post
1341
Google just dropped an exciting technical report for the brand-new Gemma3 model! 🚀 Here are my personal notes highlighting the most intriguing architectural innovations, design choices, and insights from this release:
1) Architecture choices:
> No more softcaping, replace by QK-Norm
> Both Pre AND Post Norm
> Wider MLP than Qwen2.5, ~ same depth
> SWA with 5:1 and 1024 (very small and cool ablation on the paper!)
> No MLA to save KV cache, SWA do the job!
2) Long context
> Only increase the rope in the global layer (to 1M)
> Confirmation that it's harder to do long context for smol models, no 128k for the 1B
> Pretrained with 32k context? seems very high
> No yarn nor llama3 like rope extension
3) Distillation
> Only keep te first 256 logits for the teacher
> Ablation on the teacher gap (tl;dr you need some "patience" to see that using a small teacher is better)
> On policy distillation yeahh (by
@agarwl_
et al), not sure if the teacher gap behave the same here, curious if someone have more info?
4) Others
> Checkpoint with QAT, that's very cool
> RL using improve version of BOND, WARM/WARP good excuse to look at
@ramealexandre
papers
> Only use Zero3, no TP/PP if i understand correctly ?
> Training budget relatively similar than gemma2
1) Architecture choices:
> No more softcaping, replace by QK-Norm
> Both Pre AND Post Norm
> Wider MLP than Qwen2.5, ~ same depth
> SWA with 5:1 and 1024 (very small and cool ablation on the paper!)
> No MLA to save KV cache, SWA do the job!
2) Long context
> Only increase the rope in the global layer (to 1M)
> Confirmation that it's harder to do long context for smol models, no 128k for the 1B
> Pretrained with 32k context? seems very high
> No yarn nor llama3 like rope extension
3) Distillation
> Only keep te first 256 logits for the teacher
> Ablation on the teacher gap (tl;dr you need some "patience" to see that using a small teacher is better)
> On policy distillation yeahh (by
@agarwl_
et al), not sure if the teacher gap behave the same here, curious if someone have more info?
4) Others
> Checkpoint with QAT, that's very cool
> RL using improve version of BOND, WARM/WARP good excuse to look at
@ramealexandre
papers
> Only use Zero3, no TP/PP if i understand correctly ?
> Training budget relatively similar than gemma2
- 1 reply
liked
a
model
2 days ago
updated
a
model
2 days ago
published
a
model
2 days ago
updated
a
model
2 days ago
liked
a
model
2 days ago
published
a
model
2 days ago
liked
3
models
2 days ago
updated
3
models
2 days ago
published
a
model
2 days ago
updated
a
model
2 days ago