wiki.test.raw Perplexity test results
FQ8
perplexity: tokenizing the input ..
perplexity: tokenization took 840.086 ms
perplexity: calculating perplexity over 642 chunks, n_ctx=512, batch_size=512, n_seq=1
perplexity: 0.38 seconds per pass - ETA 4.02 minutes
[1]5.0975, ..., [642]6.1787,
Final estimate: PPL = 6.1787 +/- 0.03740
llama_print_timings: total time = 217947.64 ms / 328705 tokens
Silly
perplexity: tokenizing the input ..
perplexity: tokenization took 804.012 ms
perplexity: calculating perplexity over 642 chunks, n_ctx=512, batch_size=512, n_seq=1
perplexity: 0.37 seconds per pass - ETA 3.95 minutes
[1]5.4252, ..., [642]6.3603,
Final estimate: PPL = 6.3603 +/- 0.03876
llama_print_timings: total time = 214068.87 ms / 328705 tokens
A difference of 2.89656% or close to the "Average value divergence: 2.98%"
Full outputs are here: https://rentry.co/Silly-Mistral-Perplexity
If you have a preferred test file other than wiki.test.raw, i can test that as well. (I don't know if there's a new standard or something)
Even if I don't agree with "perplexity" measurements, from what I saw by chatting with the "silly" models, they are indeed different but keep accuracy (if created with 100/20. If you want to see them crazy, create them as 100/50 or more :D
Please, do me a favour: check the MD5 of both files, because yesterday I messed up a script and it generated identical files.
Be sure they are different.
You can also use bytecomp to test: https://github.com/0wwafa/bytecomp
p.s.
add me on discord: https://discord.com/channels/@robert_46007
Please, do me a favour: check the MD5 of both files, because yesterday I messed up a script and it generated identical files.
Be sure they are different.
You can also usebytecompto test: https://github.com/0wwafa/bytecomp
Would SHA256 values work?
I don't know how to build bytecomp on windows x.x
I have to sleep now, so just incase they do work here's some of the hash tools 7Zip includes
SHA256:
Name: Mistral-7B-Instruct-v0.3.silly.gguf
Size: 7954223936 bytes (7585 MiB)
SHA256: 6c156cb6e8afa124b99296016ff369cc8d276316604a34cba3520c59486dc0ab
Name: Mistral-7B-Instruct-v0.3.fq8.gguf
Size: 7954223936 bytes (7585 MiB)
SHA256: 6d441f655d8cd1878131d6dc19327fff40d0e08b54c4d1864eb2cd8373257403
Blake2SP:
Name: Mistral-7B-Instruct-v0.3.silly.gguf
Size: 7954223936 bytes (7585 MiB)
BLAKE2sp: 67da311424b76abed0dae52d53d87a5289975e7eea8d9b67f74043b3e74f8168
Name: Mistral-7B-Instruct-v0.3.fq8.gguf
Size: 7954223936 bytes (7585 MiB)
BLAKE2sp: 6e71f647180319f7d8ff6cd12d9c3bc107409e67011207399f4b7e49e3bb50c5
& i added you on discord