Bloom-176B on Google Colab
I was wondering if there's any way to use Bloom-176B on Google Colab?
No
It's too big model for Google Colab usage limits. Even for premium or professional plans.
If inference time is not a constraint, you can run BLOOM 176-B on a desktop computer, even if you don’t have a GPU, you just need at least 16GB of RAM. The following blog post contains instructions to do this:
In my computer (i5 11gen, 16GB RAM, 1TB SSD Samsung 980 pro), the generation takes 3 minutes per token using only the CPU, which is a little slow but manageable.
I was wondering if there's any way to use Bloom-176B on Google Colab?
Found this:
https://harishgarg.com/writing/generate-content-using-bloom-open-source-ai-model/
Not sure if this helps or not.
You can run it on Runpod using KoboldAI, here (I believe you need 335GB+ of VRAM though, which isn't cheap):