cognitivecomputations

/

dolphin-2.5-mixtral-8x7b

like 1.17k

Not sure if this is something on my end or something with the model. Does this mean that the model didn't download properly for me?

RuntimeError: Cannot find any model weights with `/home/ndurkee/dolphin-2.5-mixtral-8x7b`
(RayWorkerVllm pid=2530) /usr/local/lib/python3.8/dist-packages/torch/nn/init.py:412: UserWarning: Initializing zero-element tensors is a no-op [repeated 3x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
(RayWorkerVllm pid=2530)   warnings.warn("Initializing zero-element tensors is a no-op") [repeated 3x across cluster]

Same error here

So I tested the mixtral instruct and that does work on my VLLM. I also redownloaded all the files from here. The main difference I'm seeing is that these files are .bin and the mixtral files are .safetensors. I wonder if that impacted anything. I might open an issue on the vllm github.

https://github.com/vllm-project/vllm/issues/2132

This was fixed with vllm 0.2.6. Cheers

https://github.com/vllm-project/vllm/releases/tag/v0.2.6