Dsg2/LS2.5-108M-A17M-Coder-GGUF

LS2.5: Efficientmaxxing at its finest

LS2.5 Expands upon LS2 by introducing a new 16k vocab tokenizer instead of the 8K vocab. This increases efficiency (same dataset is ~12% less tokens than 8k variant) but active parameters also increases (~3M increase).

A new 5.7B pretrain dataset was created with the following sources:

• Ultrafineweb
• Github-python
• Cosmopedia-v2
• ufw-l3
• finemath
• hermes-2.5
• ultrachat
• toolace
• hermes-fc
• glaive-v2
• heremes-single
• heremes-gl5k
• apigen-mt

LS2.5-108M-A17M-Coder

This repo contains the Coder sft variant of LS2.5-108M-A17M-base. It is additionally fine tuned on ~1B tokens of Agentic coding sft.

A 108M MoE model with top-2 experts routing, entirely trained on ~5.7B tokens with a 5060ti over ~26h.

task (acc_norm)	LS-220M-A25M	LS2-266M-A14M	LS2.5-108M-A17M-Base	LS2.5-108M-A17M-Coder	LS2.5-108M-A17M-Chat	chance
arc_easy	35.8%	36.2%	35.8%	34.4%	33.8%	25.1%
hellaswag	31.8%	30.0%	29.2%	26.6%	27.4%	25.0%
piqa	60.8%	60.0%	57.4%	58.4%	58.0%	50.0%
lambada	19.2%	19.6%	14.4%	16.6%	18.6%	0%

LS2.5-108M-A17M retains 94% of benchmark accuracy of both models flagship models while being 2.4 times smaller.

Internal benchmark:

118 Multiple choice questions and 24 open prompts was benched on LS2.5-108M-A17M-Base and the two sft variants. The multiple choice questions covered facts, common sense, following instructions, picking-on-the-topic reply, staying consistent with the conversation and simple arithmetic. Random guessing scores 25%.

The model responds to the open prompts with 4 samples + 1 greedy reply. They are checked for the following:

• ends by itself, or hit the 128 token cap
• if it mentions the topic
• if it loops
• how varied wording is
• how the reply depends on the prompt vs other topics

model	Multiple choice (length-adjusted)	Ends on its own	Loops	Wording variety	Depends on the question
LS2.5-108M-A17M-Chat	59.3%	49%	1%	0.93	+0.47
LS2.5-108M-A17M-Coder	55.9%	34%	5%	0.90	+0.40
LS2.5-108M-A17M-Base	55.9%	42%	1%	0.89	+0.38

Overall, the fine tuned models don't gain any knowledge (benchmarks pretty much unchanged) but rather their output format and reply length.

Model specs
Total parameters:	108M
Model parameters:	66M
Active parameters:	17M
Ngram table:	42M
Context length:	1024
Pretrain details
Tokens seen:	~5.7B
GPU hours (5060ti):	~26h
SFT details
Tokens seen:	~1B

Examples:

"chat" (temp 0.2)

you> hi
bot> You're welcome! If you have any other questions, feel free to ask.
      [16 tok, 1454.5 tok/s, ctx 26/4096]

you> What is the capital of France?
bot> The capital of France is Pope Francis.
      [10 tok, 1250.0 tok/s, ctx 26/4096]

you> which colour is the sky?\nA:blue\nB:green\nC:purple\nD:pink
bot> The sky is a beautiful and vibrant colors that make it look like a blue. The sky has a bright, warm glow, while the green light creates an airy atmosphere. It's also a natural beauty place to enjoy in your life.
      [48 tok, 1090.9 tok/s, ctx 85/4096]

Usage: To run this model, download the latest tinylm from github and build from source. Currently working on windows prebuilts. Then run tinylm chat LS2.5-108M-A17M-Coder-Q4_K_M.gguf 512 temp. Try temp 0.2 for general chat and 0.7 for nonsensical yap.