Usual Colab Notebook became extremely slow
Hi TheBloke,
I'm having a problem running almost every model now. I'm using the Colab notebook I have always used with success in the past (with good performances too), which is this one (https://colab.research.google.com/github/pcrii/Philo-Colab-Collection/blob/main/4bit_TextGen_Gdrive.ipynb).
Now all the models became very slow, almost unusable. Could you tell me if you have an updated Colab to refer to?
Thank you so much.
Yes I do
https://colab.research.google.com/drive/1ptEmUZkIVOv9h9N-ieLILmzzKoZZmA0s?usp=sharing
The issue is that recent versions of text-generation-webui now use AutoGPTQ, and with the old notebook you're using AutoGPTQ doesn't get compiled. So it is doing inference on the CPU, which is horribly slow. The above notebook should solve that issue.