Repository: gpustack/gpustack

Issue:
GPUStack allocated more VRAM than the available capacity
**Describe the bug**

When running the DeepSeek-R1 UD-IQ2_XXS model with 8 x 3090s, GPUStack automatically calculates and sets `--gpu-layers=49`, which means 49 layers are offloaded into VRAM, while the remaining layers are placed in RAM. However, this still results in an OOM issue.

/var/lib/gpustack/bin/llama-box_v0.0.114 --host 0.0.0.0 --embeddings --gpu-layers 49 --parallel 4 --ctx-size 8192 --port 40307 --model /home/joshua/models/DeepSeek-R1-UD-IQ2_XXS/DeepSeek-R1-UD-IQ2_XXS-00001-of-00004.gguf --alias deepseek-r1-unsloth --no-mmap --no-warmup --tensor-split 27908,23151,23151,23151,23151,23151,23151,23151

Through calculation with gguf-parser, this configuration allocates more VRAM than the capacity:

<img width="1892" alt="Image" src="https://github.com/user-attachments/assets/231dec10-71f4-4404-9661-08f1fa08072b" />

Hints from issue comments:
@Finenyaco: **GPUStack version**: main 6008e91
**llama-box version:** v0.0.114
**gguf-parser version:** v0.13.13

**Test Result**

The values of  tensor-split are uniform in main. However, inaccurate VRAM required for parser calculation leads to OOM. @thxCode 


![Image](https://github.com/user-attachments/assets/badbfa0a-afe4-4a32-9b55-96cebf12dee5)

```
2025-02-10T12:42:55+08:00 - gpustack.worker.backends.base - DEBUG - __init__ execution time: 0.03847694396972656 seconds
2025-02-10T12:42:55+08:00 - gpustack.worker.backends.llama_box - INFO - Starting llama-box server
2025-02-10T12:42:55+08:00 - gpustack.worker.backends.llama_box - DEBUG - Run llama-box: /opt/conda/lib/python3.11/site-packages/gpustack/third_party/bin/llama-box/llama-box with arguments: --host 0.0.0.0 --embeddings --gpu-layers 42 --parallel 4 --ctx-size 8192 --port 40003 --model /root/workspace/cuhjks47v38s73c5k28g/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ2_XXS/DeepSeek-R1-UD-IQ2_XXS-00001-of-00004.gguf --alias deepseek-r1 --no-mmap --no-warmup --tensor-split 23540,23540,23540,23540,23540,23540,23540
0.00.001.738 I 
0.00.001.750 I arguments  : /opt/conda/lib/python3.11/site-packages/gpustack/third_party/bin/llama-box/llama-box --host 0.0.0.0 --embeddings --gpu-layers 42 --parallel 4 --ctx-size 8192 --port 40003 --model /root/workspace/cuhjks47v38s73c5k28g/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ2_XXS/DeepSeek-R1-UD-IQ2_XXS-00001-of-00004.gguf --alias deepseek-r1 --no-mmap --no-warmup --tensor-split 23540,23540,23540,23540,23540,23540,23540
0.00.001.784 I version    : v0.0.114 (b113b73)
0.00.001.784 I compiler   : cc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)
0.00.001.784 I target     : x86_64-redhat-linux
0.00.001.785 I vendor     : llama.cpp 27d135c9 (4598), stable-diffusion.cpp 102953d (203)
0.00.041.418 I ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
0.00.041.420 I ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
0.00.041.421 I ggml_cuda_init: found 7 CUDA devices:
0.00.041.824 I   Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
0.00.042.215 I   Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
0.00.042.620 I   Device 2: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
0.00.043.017 I   Device 3: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
0.00.043.423 I   Device 4: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
0.00.043.880 I   Device 5: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
0.00.044.401 I   Device 6: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
0.00.677.782 I system_info: n_threads = 48 (n_threads_batch = 48) / 96 | CUDA : ARCHS = 600,610,700,750,800,860,890,900 | F16 = 1 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | 
0.00.677.787 I 
0.00.677.892 I srv                      main: listening, hostname = 0.0.0.0, port = 40003, n_threads = 6 + 2
0.00.679.096 I srv                      main: loading model
0.00.679.099 I srv                load_model: loading model '/root/workspace/cuhjks47v38s73c5k28g/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ2_XXS/DeepSeek-R1-UD-IQ2_XXS-00001-of-00004.gguf'
0.00.775.206 I llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) - 23432 MiB free
0.00.775.230 I llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 4090) - 23432 MiB free
0.00.775.247 I llama_model_load_from_file_impl: using device CUDA2 (NVIDIA GeForce RTX 4090) - 23432 MiB free
0.00.775.263 I llama_model_load_from_file_impl: using device CUDA3 (NVIDIA GeForce RTX 4090) - 23432 MiB free
0.00.775.280 I llama_model_load_from_file_impl: using device CUDA4 (NVIDIA GeForce RTX 4090) - 23432 MiB free
0.00.775.295 I llama_model_load_from_file_impl: using device CUDA5 (NVIDIA GeForce RTX 4090) - 23432 MiB free
0.00.775.312 I llama_model_load_from_file_impl: using device CUDA6 (NVIDIA GeForce RTX 4090) - 23432 MiB free
0.00.829.775 I llama_model_loader: additional 3 GGUFs metadata loaded.
0.00.829.785 I llama_model_loader: loaded meta data with 52 key-value pairs and 1025 tensors from /root/workspace/cuhjks47v38s73c5k28g/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ2_XXS/DeepSeek-R1-UD-IQ2_XXS-00001-of-00004.gguf (version GGUF V3 (latest))
0.00.829.817 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
0.00.829.831 I llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
0.00.829.832 I llama_model_loader: - kv   1:                               general.type str              = model
0.00.829.833 I llama_model_loader: - kv   2:                               general.name str              = DeepSeek R1 BF16
0.00.829.834 I llama_model_loader: - kv   3:                       general.quantized_by str              = Unsloth
0.00.829.835 I llama_model_loader: - kv   4:                         general.size_label str              = 256x20B
0.00.829.836 I llama_model_loader: - kv   5:                           general.repo_url str              = https://huggingface.co/unsloth
0.00.829.838 I llama_model_loader: - kv   6:                      deepseek2.block_count u32              = 61
0.00.829.838 I llama_model_loader: - kv   7:                   deepseek2.context_length u32              = 163840
0.00.829.839 I llama_model_loader: - kv   8:                 deepseek2.embedding_length u32              = 7168
0.00.829.840 I llama_model_loader: - kv   9:              deepseek2.feed_forward_length u32              = 18432
0.00.829.841 I llama_model_loader: - kv  10:             deepseek2.attention.head_count u32              = 128
0.00.829.842 I llama_model_loader: - kv  11:          deepseek2.attention.head_count_kv u32              = 128
0.00.829.847 I llama_model_loader: - kv  12:                   deepseek2.rope.freq_base f32              = 10000.000000
0.00.829.849 I llama_model_loader: - kv  13: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
0.00.829.850 I llama_model_loader: - kv  14:                deepseek2.expert_used_count u32              = 8
0.00.829.851 I llama_model_loader: - kv  15:        deepseek2.leading_dense_block_count u32              = 3
0.00.829.852 I llama_model_loader: - kv  16:                       deepseek2.vocab_size u32              = 129280
0.00.829.853 I llama_model_loader: - kv  17:            deepseek2.attention.q_lora_rank u32              = 1536
0.00.829.853 I llama_model_loader: - kv  18:           deepseek2.attention.kv_lora_rank u32              = 512
0.00.829.854 I llama_model_loader: - kv  19:             deepseek2.attention.key_length u32              = 192
0.00.829.855 I llama_model_loader: - kv  20:           deepseek2.attention.value_length u32              = 128
0.00.829.856 I llama_model_loader: - kv  21:       deepseek2.expert_feed_forward_length u32              = 2048
0.00.829.856 I llama_model_loader: - kv  22:                     deepseek2.expert_count u32              = 256
0.00.829.857 I llama_model_loader: - kv  23:              deepseek2.expert_shared_count u32              = 1
0.00.829.858 I llama_model_loader: - kv  24:             deepseek2.expert_weights_scale f32              = 2.500000
0.00.829.859 I llama_model_loader: - kv  25:              deepseek2.expert_weights_norm bool             = true
0.00.829.860 I llama_model_loader: - kv  26:               deepseek2.expert_gating_func u32              = 2
0.00.829.861 I llama_model_loader: - kv  27:             deepseek2.rope.dimension_count u32              = 64
0.00.829.861 I llama_model_loader: - kv  28:                deepseek2.rope.scaling.type str              = yarn
0.00.829.863 I llama_model_loader: - kv  29:              deepseek2.rope.scaling.factor f32              = 40.000000
0.00.829.864 I llama_model_loader: - kv  30: deepseek2.rope.scaling.original_context_length u32              = 4096
0.00.829.865 I llama_model_loader: - kv  31: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
0.00.829.865 I llama_model_loader: - kv  32:                       tokenizer.ggml.model str              = gpt2
0.00.829.866 I llama_model_loader: - kv  33:                         tokenizer.ggml.pre str              = deepseek-v3
0.00.875.274 I llama_model_loader: - kv  34:                      tokenizer.ggml.tokens arr[str,129280]  = ["<｜begin▁of▁sentence｜>", "<...
0.00.894.887 I llama_model_loader: - kv  35:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
0.00.939.630 I llama_model_loader: - kv  36:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
0.00.939.636 I llama_model_loader: - kv  37:                tokenizer.ggml.bos_token_id u32              = 0
0.00.939.637 I llama_model_loader: - kv  38:                tokenizer.ggml.eos_token_id u32              = 1
0.00.939.638 I llama_model_loader: - kv  39:            tokenizer.ggml.padding_token_id u32              = 128815
0.00.939.639 I llama_model_loader: - kv  40:               tokenizer.ggml.add_bos_token bool             = true
0.00.939.640 I llama_model_loader: - kv  41:               tokenizer.ggml.add_eos_token bool             = false
0.00.939.643 I llama_model_loader: - kv  42:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
0.00.939.645 I llama_model_loader: - kv  43:               general.quantization_version u32              = 2
0.00.939.646 I llama_model_loader: - kv  44:                          general.file_type u32              = 19
0.00.939.647 I llama_model_loader: - kv  45:                      quantize.imatrix.file str              = DeepSeek-R1.imatrix
0.00.939.648 I llama_model_loader: - kv  46:                   quantize.imatrix.dataset str              = /training_data/calibration_datav3.txt
0.00.939.649 I llama_model_loader: - kv  47:             quantize.imatrix.entries_count i32              = 720
0.00.939.650 I llama_model_loader: - kv  48:              quantize.imatrix.chunks_count i32              = 124
0.00.939.651 I llama_model_loader: - kv  49:                                   split.no u16              = 0
0.00.939.652 I llama_model_loader: - kv  50:                        split.tensors.count i32              = 1025
0.00.939.653 I llama_model_loader: - kv  51:                                split.count u16              = 4
0.00.939.654 I llama_model_loader: - type  f32:  361 tensors
0.00.939.655 I llama_model_loader: - type q2_K:   55 tensors
0.00.939.655 I llama_model_loader: - type q3_K:    3 tensors
0.00.939.655 I llama_model_loader: - type q4_K:  190 tensors
0.00.939.656 I llama_model_loader: - type q5_K:  116 tensors
0.00.939.656 I llama_model_loader: - type q6_K:  184 tensors
0.00.939.656 I llama_model_loader: - type iq2_xxs:  116 tensors
0.00.939.658 I print_info: file format = GGUF V3 (latest)
0.00.939.659 I print_info: file type   = IQ2_XXS - 2.0625 bpw
0.00.939.664 I print_info: file size   = 182.69 GiB (2.34 BPW) 
0.01.051.236 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.01.051.382 I load: special tokens cache size = 819
0.01.262.281 I load: token to piece cache size = 0.8223 MB
0.01.262.305 I print_info: arch             = deepseek2
0.01.262.306 I print_info: vocab_only       = 0
0.01.262.307 I print_info: n_ctx_train      = 163840
0.01.262.307 I print_info: n_embd           = 7168
0.01.262.307 I print_info: n_layer          = 61
0.01.262.318 I print_info: n_head           = 128
0.01.262.320 I print_info: n_head_kv        = 128
0.01.262.321 I print_info: n_rot            = 64
0.01.262.321 I print_info: n_swa            = 0
0.01.262.321 I print_info: n_embd_head_k    = 192
0.01.262.322 I print_info: n_embd_head_v    = 128
0.01.262.324 I print_info: n_gqa            = 1
0.01.262.326 I print_info: n_embd_k_gqa     = 24576
0.01.262.327 I print_info: n_embd_v_gqa     = 16384
0.01.262.328 I print_info: f_norm_eps       = 0.0e+00
0.01.262.329 I print_info: f_norm_rms_eps   = 1.0e-06
0.01.262.330 I print_info: f_clamp_kqv      = 0.0e+00
0.01.262.330 I print_info: f_max_alibi_bias = 0.0e+00
0.01.262.330 I print_info: f_logit_scale    = 0.0e+00
0.01.262.332 I print_info: n_ff             = 18432
0.01.262.332 I print_info: n_expert         = 256
0.01.262.333 I print_info: n_expert_used    = 8
0.01.262.333 I print_info: causal attn      = 1
0.01.262.333 I print_info: pooling type     = 0
0.01.262.334 I print_info: rope type        = 0
0.01.262.334 I print_info: rope scaling     = yarn
0.01.262.336 I print_info: freq_base_train  = 10000.0
0.01.262.337 I print_info: freq_scale_train = 0.025
0.01.262.337 I print_info: n_ctx_orig_yarn  = 4096
0.01.262.338 I print_info: rope_finetuned   = unknown
0.01.262.338 I print_info: ssm_d_conv       = 0
0.01.262.339 I print_info: ssm_d_inner      = 0
0.01.262.339 I print_info: ssm_d_state      = 0
0.01.262.339 I print_info: ssm_dt_rank      = 0
0.01.262.340 I print_info: ssm_dt_b_c_rms   = 0
0.01.262.340 I print_info: model type       = 671B
0.01.262.341 I print_info: model params     = 671.03 B
0.01.262.343 I print_info: general.name     = DeepSeek R1 BF16
0.01.262.343 I print_info: n_layer_dense_lead   = 3
0.01.262.343 I print_info: n_lora_q             = 1536
0.01.262.343 I print_info: n_lora_kv            = 512
0.01.262.344 I print_info: n_ff_exp             = 2048
0.01.262.344 I print_info: n_expert_shared      = 1
0.01.262.345 I print_info: expert_weights_scale = 2.5
0.01.262.345 I print_info: expert_weights_norm  = 1
0.01.262.346 I print_info: expert_gating_func   = sigmoid
0.01.262.346 I print_info: rope_yarn_log_mul    = 0.1000
0.01.262.347 I print_info: vocab type       = BPE
0.01.262.348 I print_info: n_vocab          = 129280
0.01.262.348 I print_info: n_merges         = 127741
0.01.262.350 I print_info: BOS token        = 0 '<｜begin▁of▁sentence｜>'
0.01.262.351 I print_info: EOS token        = 1 '<｜end▁of▁sentence｜>'
0.01.262.351 I print_info: EOT token        = 1 '<｜end▁of▁sentence｜>'
0.01.262.352 I print_info: PAD token        = 128815 '<｜PAD▁TOKEN｜>'
0.01.262.354 I print_info: LF token         = 201 'Ċ'
0.01.262.354 I print_info: FIM PRE token    = 128801 '<｜fim▁begin｜>'
0.01.262.355 I print_info: FIM SUF token    = 128800 '<｜fim▁hole｜>'
0.01.262.355 I print_info: FIM MID token    = 128802 '<｜fim▁end｜>'
0.01.262.356 I print_info: EOG token        = 1 '<｜end▁of▁sentence｜>'
0.01.262.356 I print_info: max token length = 256
0.16.107.526 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36172 | status 503 | cost 13817561.72s
0.16.107.639 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36174 | status 503 | cost 13817561.72s
0.16.107.659 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36176 | status 503 | cost 13817561.72s
0.16.107.671 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36178 | status 503 | cost 13817561.72s
0.17.387.015 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36180 | status 503 | cost 13817563.00s
0.20.389.408 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36182 | status 503 | cost 13817566.00s
0.23.391.471 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36184 | status 503 | cost 13817569.01s
0.26.393.230 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36186 | status 503 | cost 13817572.01s
0.27.700.974 I load_tensors: offloading 42 repeating layers to GPU
0.27.700.981 I load_tensors: offloaded 42/62 layers to GPU
0.27.700.987 I load_tensors:    CUDA_Host model buffer size = 53545.97 MiB
0.27.700.989 I load_tensors:          CPU model buffer size =   497.11 MiB
0.27.700.989 I load_tensors:        CUDA0 model buffer size = 19003.85 MiB
0.27.700.990 I load_tensors:        CUDA1 model buffer size = 19003.85 MiB
0.27.700.991 I load_tensors:        CUDA2 model buffer size = 19003.85 MiB
0.27.700.991 I load_tensors:        CUDA3 model buffer size = 19003.85 MiB
0.27.700.992 I load_tensors:        CUDA4 model buffer size = 19003.85 MiB
0.27.700.993 I load_tensors:        CUDA5 model buffer size = 19003.85 MiB
0.27.700.993 I load_tensors:        CUDA6 model buffer size = 19003.85 MiB
0.29.394.933 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36190 | status 503 | cost 13817575.01s
0.32.396.905 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36192 | status 503 | cost 13817578.01s
0.35.398.530 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36194 | status 503 | cost 13817581.01s
0.38.400.188 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36196 | status 503 | cost 13817584.02s
0.41.402.050 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36198 | status 503 | cost 13817587.02s
0.44.404.041 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36202 | status 503 | cost 13817590.02s
0.47.405.978 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36204 | status 503 | cost 13817593.02s
0.50.407.581 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36206 | status 503 | cost 13817596.02s
0.53.409.616 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36208 | status 503 | cost 13817599.02s
0.56.411.584 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36210 | status 503 | cost 13817602.03s
0.59.413.610 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36212 | status 503 | cost 13817605.03s
1.02.415.307 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36214 | status 503 | cost 13817608.03s
1.05.417.002 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36216 | status 503 | cost 13817611.03s
1.08.418.628 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36218 | status 503 | cost 13817614.03s
1.11.420.320 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36220 | status 503 | cost 13817617.04s
1.14.422.099 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36222 | status 503 | cost 13817620.04s
1.17.423.570 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36224 | status 503 | cost 13817623.04s
1.20.425.260 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36226 | status 503 | cost 13817626.04s
1.23.426.901 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36228 | status 503 | cost 13817629.04s
1.26.428.598 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36230 | status 503 | cost 13817632.04s
1.29.430.254 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36232 | status 503 | cost 13817635.05s
1.32.432.062 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36234 | status 503 | cost 13817638.05s
1.35.433.721 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36236 | status 503 | cost 13817641.05s
1.38.435.266 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36238 | status 503 | cost 13817644.05s
1.41.436.967 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36240 | status 503 | cost 13817647.05s
1.44.438.562 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36242 | status 503 | cost 13817650.05s
1.47.440.245 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36244 | status 503 | cost 13817653.06s
1.50.441.921 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36246 | status 503 | cost 13817656.06s
1.53.443.720 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36248 | status 503 | cost 13817659.06s
1.56.445.523 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36250 | status 503 | cost 13817662.06s
1.59.447.397 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36252 | status 503 | cost 13817665.06s
2.02.449.178 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36254 | status 503 | cost 13817668.06s
2.05.450.772 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36258 | status 503 | cost 13817671.07s
2.08.452.573 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36260 | status 503 | cost 13817674.07s
2.11.454.452 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36262 | status 503 | cost 13817677.07s
2.14.456.179 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36264 | status 503 | cost 13817680.07s
2.17.457.875 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36266 | status 503 | cost 13817683.07s
2.20.459.530 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36268 | status 503 | cost 13817686.07s
2.23.461.251 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36270 | status 503 | cost 13817689.08s
2.26.462.905 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36272 | status 503 | cost 13817692.08s
2.29.464.617 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36274 | status 503 | cost 13817695.08s
2.32.466.299 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36276 | status 503 | cost 13817698.08s
2.35.467.684 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36278 | status 503 | cost 13817701.08s
2.38.469.402 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36280 | status 503 | cost 13817704.08s
2.41.470.899 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36284 | status 503 | cost 13817707.09s
2.44.472.808 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36286 | status 503 | cost 13817710.09s
2.47.474.334 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36288 | status 503 | cost 13817713.09s
2.50.476.160 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36290 | status 503 | cost 13817716.09s
2.53.477.978 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36292 | status 503 | cost 13817719.09s
2.56.479.518 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36296 | status 503 | cost 13817722.09s
2.59.481.245 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36298 | status 503 | cost 13817725.10s
3.02.482.775 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36300 | status 503 | cost 13817728.10s
3.05.484.408 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36302 | status 503 | cost 13817731.10s
3.08.486.378 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36304 | status 503 | cost 13817734.10s
3.11.488.117 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36306 | status 503 | cost 13817737.10s
3.14.489.918 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36308 | status 503 | cost 13817740.11s
3.17.491.983 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36310 | status 503 | cost 13817743.11s
3.20.493.977 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36312 | status 503 | cost 13817746.11s
3.23.495.552 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36314 | status 503 | cost 13817749.11s
3.26.497.082 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36316 | status 503 | cost 13817752.11s
3.29.498.760 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36318 | status 503 | cost 13817755.11s
3.32.500.883 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36320 | status 503 | cost 13817758.12s
3.34.750.777 I llama_init_from_model: n_seq_max     = 4
3.34.750.794 I llama_init_from_model: n_ctx         = 8192
3.34.750.794 I llama_init_from_model: n_ctx_per_seq = 2048
3.34.750.794 I llama_init_from_model: n_batch       = 2048
3.34.750.795 I llama_init_from_model: n_ubatch      = 512
3.34.750.795 I llama_init_from_model: flash_attn    = 0
3.34.750.801 I llama_init_from_model: freq_base     = 10000.0
3.34.750.802 I llama_init_from_model: freq_scale    = 0.025
3.34.750.804 W llama_init_from_model: n_ctx_per_seq (2048) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
3.34.750.829 I llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0
3.35.504.005 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36322 | status 503 | cost 13817761.12s
3.38.005.529 I llama_kv_cache_init:        CPU KV buffer size = 12160.00 MiB
3.38.027.989 I llama_kv_cache_init:      CUDA0 KV buffer size =  3840.00 MiB
3.38.040.685 I llama_kv_cache_init:      CUDA1 KV buffer size =  3840.00 MiB
3.38.045.383 I llama_kv_cache_init:      CUDA2 KV buffer size =  3840.00 MiB
3.38.052.056 I llama_kv_cache_init:      CUDA3 KV buffer size =  3840.00 MiB
3.38.056.730 I llama_kv_cache_init:      CUDA4 KV buffer size =  3840.00 MiB
3.38.061.389 I llama_kv_cache_init:      CUDA5 KV buffer size =  3840.00 MiB
3.38.066.047 I llama_kv_cache_init:      CUDA6 KV buffer size =  3840.00 MiB
3.38.066.060 I llama_init_from_model: KV self size  = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB
3.38.066.090 I llama_init_from_model:        CPU  output buffer size =     0.11 MiB
3.38.071.392 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2865.50 MiB on device 0: cudaMalloc failed: out of memory
3.38.071.398 E ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 3004696576
3.38.074.064 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1180.49 MiB on device 0: cudaMalloc failed: out of memory
3.38.074.067 E ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 1237831680
3.38.076.608 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4370.04 MiB on device 0: cudaMalloc failed: out of memory
3.38.076.612 E ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 4582318080
3.38.076.613 E llama_init_from_model: failed to allocate compute buffers
3.38.505.976 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36324 | status 503 | cost 13817764.12s
3.38.810.197 E common_init_from_params: failed to create context with model '/root/workspace/cuhjks47v38s73c5k28g/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ2_XXS/DeepSeek-R1-UD-IQ2_XXS-00001-of-00004.gguf'
3.41.507.877 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36326 | status 503 | cost 13817767.12s
3.44.509.831 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36328 | status 503 | cost 13817770.13s
3.47.511.731 I srv       log_server_response: rid  | GET /v1/models 127.0.0.1:36330 | status 503 | cost 13817773.13s
3.48.514.977 E srv                load_model: failed to load model, '/root/workspace/cuhjks47v38s73c5k28g/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ2_XXS/DeepSeek-R1-UD-IQ2_XXS-00001-of-00004.gguf'
double free or corruption (out)
3.48.515.420 E srv                      main: exiting due to model loading error
```


- ngl=35

![Image](https://github.com/user-attachments/assets/cd4c4c68-fc2f-4f18-bcb1-9e451ea609c5)@thxCode: link https://github.com/gpustack/gpustack/issues/1131@thxCode: please replace gguf-parser with the latest version and test again.@Finenyaco: Verified in main f19af23, gguf-parser v0.13.14

Solution patch:
diff --git a/gpustack/policies/candidate_selectors/gguf_resource_fit_selector.py b/gpustack/policies/candidate_selectors/gguf_resource_fit_selector.py
index 0a55233..1936891 100644
--- a/gpustack/policies/candidate_selectors/gguf_resource_fit_selector.py
+++ b/gpustack/policies/candidate_selectors/gguf_resource_fit_selector.py
@@ -395,6 +395,7 @@ class GGUFResourceFitSelector(ScheduleCandidatesSelector):
                             vram=vram_claim,
                             ram=ram_claim,
                             total_layers=total_layers,
+                            tensor_split=tensor_splitting,
                         ),
                     )
                 )
@@ -626,6 +627,7 @@ class GGUFResourceFitSelector(ScheduleCandidatesSelector):
                 vram=final_gpu_claims,
                 ram=final_ram_claim,
                 total_layers=total_layers,
+                tensor_split=tensor_splitting,
             ),
         )
 
@@ -1121,6 +1123,7 @@ class GGUFResourceFitSelector(ScheduleCandidatesSelector):
                     vram=main_worker_vram_claim,
                     ram=main_worker_ram_claim,
                     total_layers=total_layers,
+                    tensor_split=flag_tensor_spliting,
                 ),
                 rpc_servers=rpc_servers,
             )
diff --git a/gpustack/scheduler/calculator.py b/gpustack/scheduler/calculator.py
index 4a585d6..34b9ceb 100644
--- a/gpustack/scheduler/calculator.py
+++ b/gpustack/scheduler/calculator.py
@@ -46,6 +46,20 @@ class memoryEstimate:
     vrams: List[layerMemoryEstimate]
     offloadLayers: Optional[int] = None  # Not available for diffusion models
 
+    def to_log_string(self) -> str:
+        vram_strings = ', '.join(
+            [
+                f"(uma:{vram.uma}, non-uma:{vram.nonuma}, layers:{vram.handleLayers})"
+                for vram in self.vrams
+            ]
+        )
+        return (
+            f"layers: {self.offloadLayers}, "
+            f"{'full offloaded, ' if self.fullOffloaded else ''}"
+            f"ram: (uma:{self.ram.uma}, non-uma:{self.ram.nonuma}, layers:{self.ram.handleLayers}), "
+            f"vrams: [{vram_strings}]"
+        )
+
 
 @dataclass_json
 @dataclass
@@ -261,7 +275,7 @@ async def calculate_model_resource_claim(
             estimate = _get_empty_estimate(n_gpu=len(tensor_split))
         return ModelInstanceResourceClaim(model_instance, estimate)
 
-    logger.info(f"Calculating resource claim for model instance {model_instance.name}")
+    logger.debug(f"Calculating resource claim for model instance {model_instance.name}")
 
     command = await _gguf_parser_command(model, offload, **kwargs)
     try:
@@ -285,18 +299,18 @@ async def calculate_model_resource_claim(
         if offload == GPUOffloadEnum.Full:
             logger.info(
                 f"Calculated resource claim for full offload model instance {model_instance.name}, "
-                f"claim: {claim.estimate.items[0]}"
+                f"{claim.estimate.items[0].to_log_string()}"
             )
         elif offload == GPUOffloadEnum.Partial:
             logger.info(
-                f"Calculated resource claim for partial offloading model instance {model_instance.name}, "
-                f"least claim: {claim.estimate.items[1]}, "
-                f"most claim: {claim.estimate.items[len(claim.estimate.items) - 2]}"
+                f"Calculated resource claim for partial offloading model instance {model_instance.name}, \n"
+                f"  Least: {claim.estimate.items[1].to_log_string()} \n"
+                f"  Most: {claim.estimate.items[len(claim.estimate.items) - 2].to_log_string()}"
             )
         elif offload == GPUOffloadEnum.Disable:
             logger.info(
                 f"Calculated resource claim for disabled offloading model instance {model_instance.name}, "
-                f"claim: {claim.estimate.items[0]}"
+                f"{claim.estimate.items[0].to_log_string()}"
             )
             clear_vram_claim(claim)
 
diff --git a/gpustack/schemas/models.py b/gpustack/schemas/models.py
index f72876e..57f7202 100644
--- a/gpustack/schemas/models.py
+++ b/gpustack/schemas/models.py
@@ -201,6 +201,7 @@ class ComputedResourceClaim(BaseModel):
     total_layers: Optional[int] = None
     ram: Optional[int] = Field(default=None)  # in bytes
     vram: Optional[Dict[int, int]] = Field(default=None)  # in bytes
+    tensor_split: Optional[List[int]] = Field(default=None)
 
 
 class ModelInstanceRPCServer(RPCServer):
diff --git a/gpustack/worker/backends/llama_box.py b/gpustack/worker/backends/llama_box.py
index c9f6244..f3baa10 100644
--- a/gpustack/worker/backends/llama_box.py
+++ b/gpustack/worker/backends/llama_box.py
@@ -40,14 +40,6 @@ class LlamaBoxServer(InferenceServer):
         if claim is not None and claim.get("offload_layers") is not None:
             layers = claim.get("offload_layers")
 
-        main_worker_tensor_split = []
-        if (
-            self._model_instance.gpu_indexes
-            and len(self._model_instance.gpu_indexes) > 0
-        ):
-            vram_claims = claim.get("vram").values()
-            main_worker_tensor_split = vram_claims
-
         workers = self._clientset.workers.list()
         worker_map = {worker.id: worker for worker in workers.items}
         rpc_servers, rpc_server_tensor_split = get_rpc_servers(
@@ -94,19 +86,34 @@ class LlamaBoxServer(InferenceServer):
             rpc_servers_argument = ",".join(rpc_servers)
             arguments.extend(["--rpc", rpc_servers_argument])
 
-        final_tensor_split = []
+        # legacy support for tensor split field is empty
+        main_worker_tensor_split = []
+        if (
+            self._model_instance.gpu_indexes
+            and len(self._model_instance.gpu_indexes) > 0
+        ):
+            vram_claims = claim.get("vram").values()
+            main_worker_tensor_split = vram_claims
+
+        legacy_tensor_split = []
         if rpc_server_tensor_split:
-            final_tensor_split.extend(rpc_server_tensor_split)
-            final_tensor_split.extend(main_worker_tensor_split)
+            legacy_tensor_split.extend(rpc_server_tensor_split)
+            legacy_tensor_split.extend(main_worker_tensor_split)
         elif len(main_worker_tensor_split) > 1:
-            final_tensor_split.extend(main_worker_tensor_split)
+            legacy_tensor_split.extend(main_worker_tensor_split)
+
+        tensor_split = legacy_tensor_split
+        if self._model_instance.computed_resource_claim.get("tensor_split"):
+            tensor_split = self._model_instance.computed_resource_claim.get(
+                "tensor_split"
+            )
 
         user_tensor_split = find_parameter(
             self._model.backend_parameters, ["ts", "tensor-split"]
         )
-        if user_tensor_split is None and final_tensor_split:
+        if user_tensor_split is None and tensor_split:
             tensor_split_argument = ",".join(
-                [str(int(tensor / (1024 * 1024))) for tensor in final_tensor_split]
+                [str(int(tensor / (1024 * 1024))) for tensor in tensor_split]
             )  # convert to MiB to prevent overflow
 
             arguments.extend(["--tensor-split", tensor_split_argument])


Test patch:
diff --git a/tests/scheduler/test_scheduler.py b/tests/scheduler/test_scheduler.py
index fa047d9..cd37434 100644
--- a/tests/scheduler/test_scheduler.py
+++ b/tests/scheduler/test_scheduler.py
@@ -244,6 +244,7 @@ async def test_schedule_to_single_worker_multi_gpu(config):
                 "ram": 455165112,
                 "vram": {0: 22912443392, 1: 22911897600},
                 "score": 58.828511417553905,
+                "tensor_split": [26015170560, 26015170560],
             },
         ]
 
@@ -316,6 +317,16 @@ async def test_schedule_to_single_worker_multi_gpu_with_deepseek_r1(config):
                     7: 21244001280,
                 },
                 "score": 100,
+                "tensor_split": [
+                    25769803776,
+                    25769803776,
+                    25769803776,
+                    25769803776,
+                    25769803776,
+                    25769803776,
+                    25769803776,
+                    25769803776,
+                ],
             },
         ]
 
@@ -415,6 +426,7 @@ async def test_schedule_to_single_worker_multi_gpu_with_binpack_spread(config):
                     2: 15703068672,
                 },
                 "score": 63.08188756952499,
+                "tensor_split": [17171480576, 17171480576, 16647192576],
             },
             {
                 "offload_layers": 81,
@@ -429,6 +441,7 @@ async def test_schedule_to_single_worker_multi_gpu_with_binpack_spread(config):
                     3: 15703068672,
                 },
                 "score": 63.40144801834418,
+                "tensor_split": [17171480576, 17171480576, 16542334976],
             },
             {
                 "offload_layers": 81,
@@ -443,6 +456,7 @@ async def test_schedule_to_single_worker_multi_gpu_with_binpack_spread(config):
                     3: 15703068672,
                 },
                 "score": 65.08312111378692,
+                "tensor_split": [17171480576, 16647192576, 16542334976],
             },
             {
                 "offload_layers": 81,
@@ -457,6 +471,7 @@ async def test_schedule_to_single_worker_multi_gpu_with_binpack_spread(config):
                     3: 15703068672,
                 },
                 "score": 65.08312111378692,
+                "tensor_split": [17171480576, 16647192576, 16542334976],
             },
         ]
 
@@ -476,18 +491,22 @@ async def test_schedule_to_single_worker_multi_gpu_with_binpack_spread(config):
             {
                 "gpu_indexes": [0, 1, 2],
                 "score": 85.0,
+                "tensor_split": [17171480576, 17171480576, 16647192576],
             },
             {
                 "gpu_indexes": [0, 1, 3],
                 "score": 85.0,
+                "tensor_split": [17171480576, 17171480576, 16542334976],
             },
             {
                 "gpu_indexes": [0, 2, 3],
                 "score": 84.0,
+                "tensor_split": [17171480576, 16647192576, 16542334976],
             },
             {
                 "gpu_indexes": [1, 2, 3],
                 "score": 84.0,
+                "tensor_split": [17171480576, 16647192576, 16542334976],
             },
         ]
 
@@ -552,6 +571,7 @@ async def test_schedule_to_single_worker_multi_gpu_partial_offload(config):
                     1: 16900820992,
                 },
                 "score": 66.15,
+                "tensor_split": [17171480576, 17171480576],
             }
         ]
 
@@ -742,6 +762,7 @@ async def test_schedule_to_multi_worker_multi_gpu(config):
                     1: 13296406528,
                 },
                 "score": 53.10511012189776,
+                "tensor_split": [26015170560, 17171480576, 17171480576],
                 "rpc_servers": [
                     ModelInstanceRPCServer(
                         worker_id=2,
@@ -767,6 +788,7 @@ async def test_schedule_to_multi_worker_multi_gpu(config):
                     0: 19475402752,
                 },
                 "score": 65.6816841222221,
+                "tensor_split": [17171480576, 17171480576, 26015170560],
                 "rpc_servers": [
                     ModelInstanceRPCServer(
                         worker_id=3,
@@ -866,6 +888,7 @@ async def test_manual_schedule_to_multi_worker_multi_gpu(config):
                     1: 13296406528,
                 },
                 "score": 53.10511012189776,
+                "tensor_split": [26015170560, 17171480576, 17171480576],
                 "rpc_servers": [
                     ModelInstanceRPCServer(
                         worker_id=2,
@@ -891,6 +914,7 @@ async def test_manual_schedule_to_multi_worker_multi_gpu(config):
                     0: 19475402752,
                 },
                 "score": 65.6816841222221,
+                "tensor_split": [17171480576, 17171480576, 26015170560],
                 "rpc_servers": [
                     ModelInstanceRPCServer(
                         worker_id=3,
@@ -995,6 +1019,14 @@ async def test_manual_schedule_to_multi_worker_multi_gpu_with_deepseek_r1(config
                     1: 68910751744,
                 },
                 "score": 100,
+                "tensor_split": [
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                ],
                 "rpc_servers": [
                     ModelInstanceRPCServer(
                         worker_id=9,
@@ -1054,6 +1086,14 @@ async def test_manual_schedule_to_multi_worker_multi_gpu_with_deepseek_r1(config
                     1: 68910751744,
                 },
                 "score": 100,
+                "tensor_split": [
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                ],
                 "rpc_servers": [
                     ModelInstanceRPCServer(
                         worker_id=8,
@@ -1113,6 +1153,14 @@ async def test_manual_schedule_to_multi_worker_multi_gpu_with_deepseek_r1(config
                     1: 68910751744,
                 },
                 "score": 100,
+                "tensor_split": [
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                    85899345920,
+                ],
                 "rpc_servers": [
                     ModelInstanceRPCServer(
                         worker_id=8,
@@ -1235,6 +1283,7 @@ async def test_manual_schedule_to_multi_worker_multi_gpu_with_deepseek_r1_distil
                     0: 16503328768,
                 },
                 "score": 100,
+                "tensor_split": [24683479040, 16106143744, 17163091968],
                 "rpc_servers": [
                     ModelInstanceRPCServer(
                         worker_id=12,
@@ -1355,13 +1404,14 @@ async def test_manual_schedule_to_single_worker_multi_gpu(config):
                     3: 15703068672,
                 },
                 "score": 65.08312111378692,
+                "tensor_split": [17171480576, 16647192576, 16542334976],
             }
         ]
 
         assert len(candidates) == 1
         compare_candidates(candidates, expected_candidates)
 
-        # update gpu selecto
+        # update gpu selector
         m.gpu_selector = GPUSelector(
             gpu_ids=["host-4-4080:cuda:0", "host-4-4080:cuda:1", "host-4-4080:cuda:2"]
         )
@@ -1385,6 +1435,7 @@ async def test_manual_schedule_to_single_worker_multi_gpu(config):
                     2: 15703068672,
                 },
                 "score": 63.08188756952499,
+                "tensor_split": [17171480576, 17171480576, 16647192576],
             }
         ]
 
@@ -1444,6 +1495,7 @@ async def test_manual_schedule_to_single_worker_multi_gpu_partial_offload(config
                     1: 16900820992,
                 },
                 "score": 66.15,
+                "tensor_split": [17171480576, 17171480576],
             }
         ]
 
@@ -1759,3 +1811,9 @@ def compare_candidates(  # noqa: C901
                     rpc_server.computed_resource_claim
                     == expected["rpc_servers"][i].computed_resource_claim
                 )
+
+        if "tensor_split" in expected:
+            assert (
+                candidate.computed_resource_claim.tensor_split
+                == expected["tensor_split"]
+            )