Why is not working? Please advice as I'm a complete beginner
Thanks a lot first of all as I am a completely beginner for AI and maybe this request is stupid or naive. After looking some tutorial on youtube about how to deploy open source models on runpod I decided to try, as it looks like not so difficult to do it, so I setup a profile and created a POD 1 x RTX A6000
16 vCPU 62 GB RAM, 1000 GB of volume. Anytime I try to download the model (but is the same with ALL the models ,so for sure the mistake is mine somewhere) , it takes a lot to complete the download and when I try to Load the model (with Transformers) the situation never change from "Loading ehartford_dolphin-2.5-mixtral-8x7b..." , and after 10 minutes nothing change so I have to stop (and money spent for nothing) without being able to try the model. Do you have an idea about where I do something wrong? As this is the same also with other models (like mistral-7b for example). Maybe the configuration is wrong? I tried to empower more the setting but nothing is changing. I'm really depressed.
it seems your network doesn't work.
Hi,
Mixtral is a huge model! To start with, I'd suggest trying one of the smaller Mistral models so you can at least confirm that you can get TextGen WebUI to load a model. This walkthrough shows the entire process and worked well for me - https://youtu.be/_59AsSyMERQ?si=C1aEF5TLr7Ymb6ts