Repository: oobabooga/text-generation-webui

Issue:
Problem when setting CPU memory limit
### Describe the bug

When trying to set a CPU memory limit, I get this error:

line 85, in convert_file_size_to_int

raise ValueError(err_msg)
ValueError: size 64000MiBGiB is not in a valid format. Use an integer for bytes, or a string with an unit (like ‘5.0GB’).

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

Change the CPU memory and try to load a model

### Screenshot

<img width="958" alt="image" src="https://github.com/oobabooga/text-generation-webui/assets/128736804/689f1d6e-fa77-4598-a1f6-4c55eb47bdac">


### Logs

```shell
Traceback (most recent call last):

File “/home/chungus/Downloads/LocalGPT/installer_files/env/lib/python3.10/site-packages/accelerate/utils/modeling.py”, line 70, in convert_file_size_to_int

mem_size = int(float(size[:-3]) * (2**30))
ValueError: could not convert string to float: ‘64000MiB’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File “/home/chungus/Downloads/LocalGPT/text-generation-webui/modules/ui_model_menu.py”, line 195, in load_model_wrapper

shared.model, shared.tokenizer = load_model(shared.model_name, loader)
File “/home/chungus/Downloads/LocalGPT/text-generation-webui/modules/models.py”, line 79, in load_model

output = load_func_map[loader](model_name)
File “/home/chungus/Downloads/LocalGPT/text-generation-webui/modules/models.py”, line 320, in AutoGPTQ_loader

return modules.AutoGPTQ_loader.load_quantized(model_name)
File “/home/chungus/Downloads/LocalGPT/text-generation-webui/modules/AutoGPTQ_loader.py”, line 57, in load_quantized

model = AutoGPTQForCausalLM.from_quantized(path_to_model, **params)
File “/home/chungus/Downloads/LocalGPT/installer_files/env/lib/python3.10/site-packages/auto_gptq/modeling/auto.py”, line 108, in from_quantized

return quant_func(
File “/home/chungus/Downloads/LocalGPT/installer_files/env/lib/python3.10/site-packages/auto_gptq/modeling/_base.py”, line 859, in from_quantized

max_memory = accelerate.utils.get_balanced_memory(
File “/home/chungus/Downloads/LocalGPT/installer_files/env/lib/python3.10/site-packages/accelerate/utils/modeling.py”, line 767, in get_balanced_memory

max_memory = get_max_memory(max_memory)
File “/home/chungus/Downloads/LocalGPT/installer_files/env/lib/python3.10/site-packages/accelerate/utils/modeling.py”, line 654, in get_max_memory

max_memory[key] = convert_file_size_to_int(max_memory[key])
File “/home/chungus/Downloads/LocalGPT/installer_files/env/lib/python3.10/site-packages/accelerate/utils/modeling.py”, line 85, in convert_file_size_to_int

raise ValueError(err_msg)
ValueError: size 64000MiBGiB is not in a valid format. Use an integer for bytes, or a string with an unit (like ‘5.0GB’).
```


### System Info

```shell
Most recent Ubuntu, RTX 4070 (gpu0) and RTX 3070 (gpu1)
```



Discussion:
*I'm in the same boat with the same problem.

 I just avoided using CPU regarding sliders
This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.

Resolving patch:
diff --git a/modules/models.py b/modules/models.py
--- a/modules/models.py
+++ b/modules/models.py
@@ -381,15 +381,13 @@ def RWKV_loader(model_name):
 
 
 def get_max_memory_dict():
-    max_memory = {}
+    max_cpu_memory = shared.args.cpu_memory.strip() if shared.args.cpu_memory is not None else '99GiB'
+    max_memory = {'cpu': f'{max_cpu_memory}GiB' if not re.match('.*ib$', max_cpu_memory.lower()) else max_cpu_memory}
     if shared.args.gpu_memory:
         memory_map = list(map(lambda x: x.strip(), shared.args.gpu_memory))
         for i in range(len(memory_map)):
             max_memory[i] = f'{memory_map[i]}GiB' if not re.match('.*ib$', memory_map[i].lower()) else memory_map[i]
 
-        max_cpu_memory = shared.args.cpu_memory.strip() if shared.args.cpu_memory is not None else '99GiB'
-        max_memory['cpu'] = f'{max_cpu_memory}GiB' if not re.match('.*ib$', max_cpu_memory.lower()) else max_cpu_memory
-
     # If --auto-devices is provided standalone, try to get a reasonable value
     # for the maximum memory of device :0
     elif shared.args.auto_devices:
@@ -403,7 +401,7 @@ def get_max_memory_dict():
 
         suggestion = int(round(suggestion / 1000))
         logger.warning(f"Auto-assiging --gpu-memory {suggestion} for your GPU to try to prevent out-of-memory errors. You can manually set other values.")
-        max_memory = {0: f'{suggestion}GiB', 'cpu': f'{shared.args.cpu_memory or 99}GiB'}
+        max_memory[0] = f'{suggestion}GiB'
 
     return max_memory if len(max_memory) > 0 else None
 


Test patch: