GPT-J to ggml conversion help
Hi, I'm looking to convert a gpt-j based model into ggml but I can't seem to find a good source to learn how to do it.
This is probably the place to start: https://github.com/ggerganov/ggml/tree/master/examples/gpt-j
There is a convert script but I believe it's intended to be used on the big h5 format base model. Can you share some tips on conversion, thanks.
Sorry, looks like that's actually the script we should use. Here are a couple more as well: https://github.com/NolanoOrg/cformers/tree/master/cformers/cpp/converters
I use https://github.com/ggerganov/ggml/ you need the full model and run the convert script on examples/gpt-j/convert-h5-to-ggml.py to generate a f32 or f16 model. Then you quantize the f32 or f16 using /build/bin/gpt-j-quantize
You need to compile following the instructions on the repo.
Thanks for the reply. I was actually following that workflow on WSL but for some reason the script is killed while loading model.state_dict(). There is no helpful error message but most likely it's a memory issue, I have 32gb but Windows is probably eating up too much for this stuff.