FYI -TG WebUI - Ubuntu with Nvidia 4090 & ExLlama_HF
Thanks @deepseek-coder & @TheBloke - What an impressive Model
FYI to those looking to run on Nvidia 4090 using ExLlama_HF I had to modify "max_seq_len" to 10000
Then memory was not an issue but still an impressive context window
Thanks to all
I have still yet to adapt things to work ( not a mac here, i have other issues to work thru, including some cuda version issues ). Did you ever use wizard python ? if so, how do the 2 compare? ( to decide if this worth it to me or not. So far wizard python is the best i have found, and that is my only use case, Python code )
i'm super impressed - i've been using it to write scripts add capabilities to nautilus on linux
i'd previously used GPT for this and it sent me on a wild goose chase. this model seems more precise
re python, i got it to write me a tool for renaming files with various features in a UI.
it did much of it pretty seamlessly and then we enhanced it to handle various other scenarios.
as for comparison to other models...it's the first model where i'm feeling comfortable leaving GPT to the side though I did get it to score some scripts
I will try and spend more time on phind & wizardcoder to do a better comparison
sounds like i should spend the time getting it to work. i moved from GPT to wizardpython when it first came out. Not looked back. But i do try other open models as they happen and so far, nothing has beat it for me.
( my problem is due to the ever changing needs of the models for GGUF/CPU, and seems some of the libraries now dont like my GPU driver version on another machine, not that it cant be fixed, just didnt want to take the effort unless it was worth it )
Sounds like I need to spend some more time with Wizard - Phind is quite nice too
Yea I'm having problems with the GGUF model on Mac M2, M1 is fine and I can't figure out why!
I even matched the env package versions to no avail
I copied over the conda env from M1 to M2 and it's working (CPU only)...head scratcher