can't run the model using vllm
#4
by
agoudarzi
- opened
when I try this model with vllm I this error:
return engine_class(*args, **kwargs)
File "/home/codespace/vllm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 110, in __init__
self.model_executor = executor_class(model_config, cache_config,
File "/home/codespace/vllm/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 37, in __init__
self._init_worker()
File "/home/codespace/vllm/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 66, in _init_worker
self.driver_worker.load_model()
File "/home/codespace/vllm/lib/python3.10/site-packages/vllm/worker/worker.py", line 107, in load_model
self.model_runner.load_model()
File "/home/codespace/vllm/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 95, in load_model
self.model = get_model(
File "/home/codespace/vllm/lib/python3.10/site-packages/vllm/model_executor/model_loader.py", line 91, in get_model
model = model_class(model_config.hf_config, linear_method)
File "/home/codespace/vllm/lib/python3.10/site-packages/vllm/model_executor/models/mpt.py", line 257, in __init__
self.transformer = MPTModel(config, linear_method)
File "/home/codespace/vllm/lib/python3.10/site-packages/vllm/model_executor/models/mpt.py", line 208, in __init__
assert config.embedding_fraction == 1.0
AssertionError
Dear @agoudarzi ,
As SEA-LION architecture is not natively supported by vLLM, additional adaptation is required for SEA-LION to run on vLLM.
Kindly refer to the readme here for the instructions on running SEA-LION with vLLM.
https://github.com/aisingapore/sealion/tree/vllm/vllm
Hope this helps.
Raymond