Benchmark validity?
I have compared Ornith 1.0 35B A3B in APEX I-Quality quantization with GRM-3.2-Sky in Q4_K_M quantization in Benchlocal with 9x runs. The same sampling parameters were used for both:
jinja = true
reasoning-preserve = true
temp = 0.6
top-p = 0.95
top-k = 20
repeat-penalty = 1.05
b = 16384
ub = 2048
HermesAgent-20:
Ornith 1.0 35B A3B: 88
GRM-3.2-Sky: 83
BugFind-15:
Ornith 1.0 35B A3B: 92
GRM-3.2-Sky: 85
Note that this is an unfair comparison since Ornith is using a superior quaint, but I don't have the hardware to quantize 35B models currently. Will update if someone releases APEX quantizations for this model. Still, the fact that GRM-3.2-Sky, which is supposedly superior to Ornith-1.0-35B according to benchmarks, is worse than it even at Q4_K_M is suspicious and brings the validity of GRM-3.2-Sky's benchmarks into question.
im making some of my own quants- its based on apex- but i modified it slightly
ill comment the link here when theyre done cooking!
Just curious, can you quantize a 35B model (~70gb) if you only gave 32GB ram?
Just curious, can you quantize a 35B model (~70gb) if you only gave 32GB ram?
you can- I did but on 12gb
just takes a painstaking amount of python orchestration- around like 2000 lines
I might open source- but im a perfectionist so idk when ill release it
I do quantize models I like with it- so just @ me and ill consider giving it a quant!
I'll post my results in a few hours. HermesAgent-20 and BugFind-15 are the main ones I care about, but if you want I can run others. I have a R9700 AI PRO, so I will run it in Quality mode. (though I use Q8 KV if possible so I might need to reduce ctx size to like 200k)
I'll post my results in a few hours. HermesAgent-20 and BugFind-15 are the main ones I care about, but if you want I can run others. I have a R9700 AI PRO, so I will run it in Quality mode. (though I use Q8 KV if possible so I might need to reduce ctx size to like 200k)
Is there the update?
el4's quaint ended up being broken, so I'm waiting until it is fixed.
working on the fix right now
my refactor broke the mapping- a fix should be either ready today or tmrw