databricks

/

dolly-v2-7b

like 146

Hello

I am on the dolly journey

I started testing this on a macbook and got to 1h40 per response.
Then moved to a dedicated server with a random GPU with 8 core ... 20 minutes
Now I am in google cloud on a NVDIA T4 16 vCPU with 60GB memory .. I got it down to 4 minutes per answer

my goal is to get to a few secs, what would be your recommendations ? I read I need a NVDIA A100 but I can't provision that on google cloud (not a single available) not on AWS (they decline the request all the time)

Thanks for your help !