Repository: vllm-project/vllm
Issue category: Feature Request

Issue:
[Bug]:  Support Falcon Mamba 
### Your current environment

Does VLLM support  Falcon Mamba models? if not, when it will be supported 

### 🐛 Describe the bug

Does VLLM support  Falcon Mamba models? if not, when it will be supported 


Hints:
cc @tlrmchlsmth 
Unsubscribe

On Wed, 14 Aug, 2024, 1:37 am Robert Shaw, ***@***.***> wrote:

> cc @tlrmchlsmth <https://github.com/tlrmchlsmth>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/vllm-project/vllm/issues/7478#issuecomment-2287037438>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BCU643RS6ZZAY5MXB4JRVXLZRJRR3AVCNFSM6AAAAABMO6JHLOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBXGAZTONBTHA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

Hey @hahmad2008, 

No, vLLM doesn't support Falcon Mamba yet. I have a work-in-progress PR (https://github.com/vllm-project/vllm/pull/6484) to add support for Mamba. I'll look into supporting FalconMamba as well.
Thank @tlrmchlsmth. Do you have any idea when this PR will be merged?
@tlrmchlsmth Do you have any idea when this PR will be merged?
Hi @hahmad2008, I’ve been prioritizing that PR over the last couple of days and I think will land it later this week. Do note that I’m not planning to add FalconMamba to #6484, but if it’s similar enough to Mamba or Mamba2, there will be a fast follow PR for support 
Thank you @tlrmchlsmth. I can only load and apply inference from it using transformers version `4.45.0.dev0` which is still not released. I installed it using: 
```
pip install -U git+https://github.com/huggingface/transformers.git 
```
So do you think your PR handle this?
@hahmad2008 sorry, haven't gotten a chance to look at FalconMamba yet -- If transformers 4.45 is needed, then I'll likely I'll wait for that release.
@tlrmchlsmth Thanks! seems it will be released next week.
https://github.com/huggingface/transformers/issues/33236#issuecomment-2324529754
i think now it is released. @tlrmchlsmth please update on this.
I'll be landing Mamba soon, possibly today. Sorry for the delay on that one. 

FalconMamba looks to be quite close to Mamba, so should be fairly easy to support. I'll try to get to it when I have some spare cycles but can't promise that I'll prioritize it. If somebody else wants to work on FalconMamba before I get to it, I would be happy to shepherd it
Hey Guys, 
 i managed to add support for FalconMamba inside vllm , 
 here is the link: #9325 
 Please feel free to check that out.
Thank you. I will try deploying it.
I am getting error when i try to deploy. Below are the steps followed:
- I followed https://docs.vllm.ai/en/stable/getting_started/installation.html#full-build-with-compilation to build fully
- I run the server using `vllm serve tiiuae/falcon-mamba-7b-instruct --dtype auto` command. I tried both with `-tp 4` and without `-tp 4`

**Hardware:**

AWS machine: g5.x24large (4xA10g)

I am getting below error:
```
INFO:     Uvicorn running on socket ('0.0.0.0', 8000) (Press CTRL+C to quit)
ERROR 10-16 12:19:35 engine.py:160] ZeroDivisionError('division by zero')
ERROR 10-16 12:19:35 engine.py:160] Traceback (most recent call last):
ERROR 10-16 12:19:35 engine.py:160]   File "/workspace/vllm/vllm/engine/multiprocessing/engine.py", line 158, in start
ERROR 10-16 12:19:35 engine.py:160]     self.run_engine_loop()
ERROR 10-16 12:19:35 engine.py:160]   File "/workspace/vllm/vllm/engine/multiprocessing/engine.py", line 214, in run_engine_loop
ERROR 10-16 12:19:35 engine.py:160]     self.engine.do_log_stats()
ERROR 10-16 12:19:35 engine.py:160]   File "/workspace/vllm/vllm/engine/llm_engine.py", line 1543, in do_log_stats
ERROR 10-16 12:19:35 engine.py:160]     stats = self._get_stats(scheduler_outputs, model_output,
ERROR 10-16 12:19:35 engine.py:160]   File "/workspace/vllm/vllm/engine/llm_engine.py", line 1583, in _get_stats
ERROR 10-16 12:19:35 engine.py:160]     gpu_cache_usage_sys = 1.0 - (num_free_gpu / num_total_gpu)
ERROR 10-16 12:19:35 engine.py:160] ZeroDivisionError: division by zero
INFO 10-16 12:19:35 multiproc_worker_utils.py:134] Terminating local vLLM worker processes
(VllmWorkerProcess pid=8683) INFO 10-16 12:19:35 multiproc_worker_utils.py:242] Worker exiting
(VllmWorkerProcess pid=8684) INFO 10-16 12:19:35 multiproc_worker_utils.py:242] Worker exiting
(VllmWorkerProcess pid=8685) INFO 10-16 12:19:35 multiproc_worker_utils.py:242] Worker exiting
ERROR 10-16 12:19:45 client.py:250] TimeoutError('No heartbeat received from MQLLMEngine')
ERROR 10-16 12:19:45 client.py:250] NoneType: None
```

@dhiaEddineRhaiem What could be the issue?
Hello @hassanraha , 

  I got the same error even with mamba and not only with FalconMamba. So it is independant on the PR implementation.

  As a workaround , i hardcoded it to the total number of gpus i have locally in my ec2 instance. 
  
 

@dhiaEddineRhaiem thanks. I changed `num_total_gpu` to number of GPU available. It worked.
@tlrmchlsmth is the below issue also fixed?
```
INFO:     Uvicorn running on socket ('0.0.0.0', 8000) (Press CTRL+C to quit)
ERROR 10-16 12:19:35 engine.py:160] ZeroDivisionError('division by zero')
ERROR 10-16 12:19:35 engine.py:160] Traceback (most recent call last):
ERROR 10-16 12:19:35 engine.py:160]   File "/workspace/vllm/vllm/engine/multiprocessing/engine.py", line 158, in start
ERROR 10-16 12:19:35 engine.py:160]     self.run_engine_loop()
ERROR 10-16 12:19:35 engine.py:160]   File "/workspace/vllm/vllm/engine/multiprocessing/engine.py", line 214, in run_engine_loop
ERROR 10-16 12:19:35 engine.py:160]     self.engine.do_log_stats()
ERROR 10-16 12:19:35 engine.py:160]   File "/workspace/vllm/vllm/engine/llm_engine.py", line 1543, in do_log_stats
ERROR 10-16 12:19:35 engine.py:160]     stats = self._get_stats(scheduler_outputs, model_output,
ERROR 10-16 12:19:35 engine.py:160]   File "/workspace/vllm/vllm/engine/llm_engine.py", line 1583, in _get_stats
ERROR 10-16 12:19:35 engine.py:160]     gpu_cache_usage_sys = 1.0 - (num_free_gpu / num_total_gpu)
ERROR 10-16 12:19:35 engine.py:160] ZeroDivisionError: division by zero
INFO 10-16 12:19:35 multiproc_worker_utils.py:134] Terminating local vLLM worker processes
(VllmWorkerProcess pid=8683) INFO 10-16 12:19:35 multiproc_worker_utils.py:242] Worker exiting
(VllmWorkerProcess pid=8684) INFO 10-16 12:19:35 multiproc_worker_utils.py:242] Worker exiting
(VllmWorkerProcess pid=8685) INFO 10-16 12:19:35 multiproc_worker_utils.py:242] Worker exiting
ERROR 10-16 12:19:45 client.py:250] TimeoutError('No heartbeat received from MQLLMEngine')
ERROR 10-16 12:19:45 client.py:250] NoneType: None
```

Patch:
diff --git a/vllm/engine/llm_engine.py b/vllm/engine/llm_engine.py
index 3a29e6a9ae094..51a0d10db8f38 100644
--- a/vllm/engine/llm_engine.py
+++ b/vllm/engine/llm_engine.py
@@ -1612,7 +1612,7 @@ def _get_stats(self,
         # KV Cache Usage in %
         num_total_gpu = self.cache_config.num_gpu_blocks
         gpu_cache_usage_sys = 0.
-        if num_total_gpu is not None:
+        if num_total_gpu:  # Guard against both None and 0
             num_free_gpu = sum(
                 scheduler.block_manager.get_num_free_gpu_blocks()
                 for scheduler in self.scheduler)
@@ -1620,7 +1620,7 @@ def _get_stats(self,
 
         num_total_cpu = self.cache_config.num_cpu_blocks
         cpu_cache_usage_sys = 0.
-        if num_total_cpu is not None and num_total_cpu > 0:
+        if num_total_cpu:  # Guard against both None and 0
             num_free_cpu = sum(
                 scheduler.block_manager.get_num_free_cpu_blocks()
                 for scheduler in self.scheduler)


Test patch: