Transformers
GGUF
English
mistral
Generated from Trainer

Possible Loading Error with GPT4All

#1
by deleted - opened

It may just be on my end, but this model doesn't load for me in GPT4All. I've tried about a dozen other recent GGUF models and they all work. It says it's an "invalid format".

Note: I tried both the Q4 0 and Q4 K M versions.

Same problem here. candle tells me it's in GGUF version 3, but I assume version 2 is expected.

bit off topic: what's the performance difference compared to llamacpp? @Phil337

deleted

I don't know much about how all this works. From what I read GPT4All uses a fork of llamacpp. Whenever I come across a bad response I test it on the full unquantized version online and it gives comparable outputs and makes the same mistakes, so there's little difference in terms of hallucinations and other errors. And in terms of speed it's about 3 tokens/second using 3 cores at 2.7 ghz with AVX2 CPU instructions. @qm9

deleted

Update: It seems all the new Bloke GGUF models are now incompatible with GPT4ALL. Earlier releases all worked (such as Dolphin 2.1 and Open Hermes 2), but all subsequent releases don't (just tested AshhLimaRP). Perhaps there was a recent change to the standard and the newer models will work with a future version of GPT4All.

There's actually no real difference in the file format except the version bump as far as I know (except if you happen to be on a big-endian platform which is unlikely), so you can just change the 5th byte (index 4) of the file from 0x03 to 0x02 to transform the GGMLv3 file to a GGMLv2 one. Some examples of how to edit a byte like that: https://stackoverflow.com/a/34524796

deleted

Thanks @KerfuffleV2 , I bet you're right since this LLM works with other apps, earlier Mistrals all worked in GPT4ALL, and it fails immediately before even trying to load a LLM into RAM. So detecting an unsupported format (GGUFv3), then refusing to load it, adds up.

I bet you're right since this LLM works with other apps

I actually checked the patch where the GGUF version changed to 3 to make sure as well, so I was pretty confident about it.

I just looked at the GPT4ALL project and it seems like they already pulled in the changes for supporting GGUFv3 files, so if you use the main branch the GGUFv3 models should work. Otherwise, I guess wait for them to make a release (or use the model revert workaround I suggested). The good news is it shouldn't be a pain point for much longer.

deleted

Cool. I'll wait. It's nice to know it's already been addressed.

What ver of GPT4All are you using ?

deleted

I'm using GPT4All v2.5.1.

Ok thx. Good to know coz i was just about to download the model and got the same ver of gpt4all. : )

Sign up or log in to comment