Repository: InftyAI/llmaz
Pull request #402: feat: support speculative decoding for llamacpp
Author: login: cr7258
Labels: name: feature
description: Categorizes issue or PR as related to a new feature.
name: lgtm
description: Looks good to me, indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: needs-priority
description: Indicates a PR lacks a  label and requires one.
name: triage/accepted
description: Indicates an issue or PR is ready to be actively worked on.
Created: 2025-05-06T14:04:58Z | Merged: 2025-05-09T05:54:30Z

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What this PR does / why we need it

Support speculative decoding for llama.cpp, which significantly improves response latency by leveraging draft model predictions. From the logs, we can see the main model and draft model are loaded successfully.


```bash
kubectl logs llamacpp-speculator-0 

Defaulted container "model-runner" out of: model-runner, model-loader (init), model-loader-1 (init)
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
build: 5280 (27aa2595) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
system info: n_threads = 4, n_threads_batch = 4, total_threads = 8

system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | 

main: binding port with default address family
main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 7
main: loading model

# load main model
srv    load_model: loading model '/workspace/models/llama-2-7b.Q8_0.gguf'
llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from /workspace/models/llama-2-7b.Q8_0.gguf (version GGUF V2)
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 7
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "
", "
", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  18:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q8_0:  226 tensors
print_info: file format = GGUF V2
print_info: file type   = Q8_0
print_info: file size   = 6.67 GiB (8.50 BPW) 
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 3
load: token to piece cache size = 0.1684 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 4096
print_info: n_embd           = 4096
print_info: n_layer          = 32
print_info: n_head           = 32
print_info: n_head_kv        = 32
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: n_swa_pattern    = 1
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 4096
print_info: n_embd_v_gqa     = 4096
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 11008
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 4096
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 7B
print_info: model params     = 6.74 B
print_info: general.name     = LLaMA v2
print_info: vocab type       = SPM
print_info: n_vocab          = 32000
print_info: n_merges         = 0
print_info: BOS token        = 1 '
'
print_info: EOS token        = 2 '
'
print_info: UNK token        = 0 '<unk>'
print_info: LF token         = 13 '<0x0A>'
print_info: EOG token        = 2 '
'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors:   CPU_Mapped model buffer size =  6828.64 MiB
...................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 1
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context:        CPU  output buffer size =     0.12 MiB
llama_kv_cache_unified: kv_size = 4096, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1, padding = 256
llama_kv_cache_unified:        CPU KV buffer size =  2048.00 MiB
llama_kv_cache_unified: KV self size  = 2048.00 MiB, K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
llama_context:        CPU compute buffer size =    71.01 MiB
llama_context: graph nodes  = 967
llama_context: graph splits = 1
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)

# load draft model
srv    load_model: loading draft model '/workspace/models/llama-2-7b.Q2_K.gguf'
llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from /workspace/models/llama-2-7b.Q2_K.gguf (version GGUF V2)
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 10
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "
", "
", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  18:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q2_K:   65 tensors
llama_model_loader: - type q3_K:  160 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V2
print_info: file type   = Q2_K - Medium
print_info: file size   = 2.63 GiB (3.35 BPW) 
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 3
load: token to piece cache size = 0.1684 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 4096
print_info: n_embd           = 4096
print_info: n_layer          = 32
print_info: n_head           = 32
print_info: n_head_kv        = 32
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: n_swa_pattern    = 1
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 4096
print_info: n_embd_v_gqa     = 4096
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 11008
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 4096
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 7B
print_info: model params     = 6.74 B
print_info: general.name     = LLaMA v2
print_info: vocab type       = SPM
print_info: n_vocab          = 32000
print_info: n_merges         = 0
print_info: BOS token        = 1 '
'
print_info: EOS token        = 2 '
'
print_info: UNK token        = 0 '<unk>'
print_info: LF token         = 13 '<0x0A>'
print_info: EOG token        = 2 '
'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors:   CPU_Mapped model buffer size =  2694.32 MiB
.................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 1
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context:        CPU  output buffer size =     0.12 MiB
llama_kv_cache_unified: kv_size = 4096, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1, padding = 256
llama_kv_cache_unified:        CPU KV buffer size =  2048.00 MiB
llama_kv_cache_unified: KV self size  = 2048.00 MiB, K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
llama_context:        CPU compute buffer size =    71.01 MiB
llama_context: graph nodes  = 967
llama_context: graph splits = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 1
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 4096
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 1
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context:        CPU  output buffer size =     0.12 MiB
llama_kv_cache_unified: kv_size = 4096, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1, padding = 256
llama_kv_cache_unified:        CPU KV buffer size =  2048.00 MiB
llama_kv_cache_unified: KV self size  = 2048.00 MiB, K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
llama_context:        CPU compute buffer size =    71.01 MiB
llama_context: graph nodes  = 967
llama_context: graph splits = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 4096
main: model loaded
main: chat template, chat_template: {%- for message in messages -%}
  {{- '
' + message.role + '
' + message.content + '

' -}}
{%- endfor -%}
{%- if add_generation_prompt -%}
  {{- '
assistant
' -}}
{%- endif -%}, example_format: '
system
You are a helpful assistant


user
Hello


assistant
Hi there


user
How are you?


assistant
'
main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  log_server_r: request: GET /health 240.243.170.78 200
```

Send a inference request:

```bash
curl --request POST \
    --url http://localhost:8080/v1/completions \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'
{"choices":[{"text":"\n1. Register a domain name (if you don’t have one yet)\n2. Choose a web host\n3. Choose a WordPress theme\n4. Create your website (using WordPress)\n5. Create and upload your website\n6. Optimize your website for search engines\n8. Test your website\n9. Monitor your website\n10. Update your website\nBuilding a website is not difficult if you have the right tools and resources.\nIn this article, we’ll cover everything you need to know to get started, including choosing a domain name, finding a web","index":0,"logprobs":null,"finish_reason":"length"}],"created":1746539608,"model":"gpt-3.5-turbo","system_fingerprint":"b5280-27aa2595","object":"text_completion","usage":{"completion_tokens":128,"prompt_tokens":14,"total_tokens":142},"id":"chatcmpl-mNfNdrU0Kivf873PYZyw0R5Ahhi1KNnX","timings":{"prompt_n":1,"prompt_ms":194.28,"prompt_per_token_ms":194.28,"prompt_per_second":5.147210212065061,"predicted_n":128,"predicted_ms":35572.597,"predicted_per_token_ms":277.9109140625,"predicted_per_second":3.598275380344033,"draft_n":155,"draft_n_accepted":15}}
```

#### Which issue(s) this PR fixes
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes https://github.com/InftyAI/llmaz/issues/240

#### Special notes for your reviewer

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
-->
```release-note
support speculative decoding for llamacpp
```



Diff:
diff --git a/api/inference/v1alpha1/config_types.go b/api/inference/v1alpha1/config_types.go
index de720569..a51a2314 100644
--- a/api/inference/v1alpha1/config_types.go
+++ b/api/inference/v1alpha1/config_types.go
@@ -42,7 +42,6 @@ type BackendRuntimeConfig struct {
 	// ConfigName represents the recommended configuration name for the backend,
 	// It will be inferred from the models in the runtime if not specified, e.g. default,
 	// speculative-decoding.
-	// +kubebuilder:default=default
 	ConfigName *string `json:"configName,omitempty"`
 	// Args defined here will "append" the args defined in the recommendedConfig,
 	// either explicitly configured in configName or inferred in the runtime.
diff --git a/chart/templates/backends/llamacpp.yaml b/chart/templates/backends/llamacpp.yaml
index fb6ae6d0..3fdada9c 100644
--- a/chart/templates/backends/llamacpp.yaml
+++ b/chart/templates/backends/llamacpp.yaml
@@ -49,6 +49,20 @@ spec:
         limits:
           cpu: 2
           memory: 4Gi
+    - name: speculative-decoding
+      args:
+        - -m
+        - "{{`{{ .ModelPath }}`}}"
+        - -md
+        - "{{`{{ .DraftModelPath }}`}}"
+        - --host
+        - "0.0.0.0"
+        - --port
+        - "8080"
+        - --draft-max
+        - "16"
+        - --draft-min
+        - "5"
   startupProbe:
     periodSeconds: 10
     failureThreshold: 30
diff --git a/config/crd/bases/inference.llmaz.io_playgrounds.yaml b/config/crd/bases/inference.llmaz.io_playgrounds.yaml
index e5854951..00ec2e20 100644
--- a/config/crd/bases/inference.llmaz.io_playgrounds.yaml
+++ b/config/crd/bases/inference.llmaz.io_playgrounds.yaml
@@ -59,7 +59,6 @@ spec:
                       the hood, e.g. vLLM.
                     type: string
                   configName:
-                    default: default
                     description: |-
                       ConfigName represents the recommended configuration name for the backend,
                       It will be inferred from the models in the runtime if not specified, e.g. default,
diff --git a/docs/examples/README.md b/docs/examples/README.md
index 6733d004..42a745b4 100644
--- a/docs/examples/README.md
+++ b/docs/examples/README.md
@@ -12,6 +12,7 @@ We provide a set of examples to help you serve large language models, by default
 - [Deploy models via TensorRT-LLM](#deploy-models-via-tensorrt-llm)
 - [Deploy models via text-generation-inference](#deploy-models-via-text-generation-inference)
 - [Deploy models via ollama](#deploy-models-via-ollama)
+- [Speculative Decoding with llama.cpp](#speculative-decoding-with-llamacpp)
 - [Speculative Decoding with vLLM](#speculative-decoding-with-vllm)
 - [Multi-Host Inference](#multi-host-inference)
 - [Deploy Host Models](#deploy-host-models)
@@ -59,6 +60,10 @@ By default, we use [vLLM](https://github.com/vllm-project/vllm) as the inference
 
 [ollama](https://github.com/ollama/ollama) based on llama.cpp, aims for local deploy. see [example](./ollama/) here.
 
+### Speculative Decoding with llama.cpp
+
+llama.cpp supports speculative decoding to significantly improve inference performance, see [example](./speculative-decoding/llamacpp/) here.
+
 ### Speculative Decoding with vLLM
 
 [Speculative Decoding](https://arxiv.org/abs/2211.17192) can improve inference performance efficiently, see [example](./speculative-decoding/vllm/) here.
diff --git a/docs/examples/speculative-decoding/llamacpp/playground.yaml b/docs/examples/speculative-decoding/llamacpp/playground.yaml
index a90b44b4..a02fccda 100644
--- a/docs/examples/speculative-decoding/llamacpp/playground.yaml
+++ b/docs/examples/speculative-decoding/llamacpp/playground.yaml
@@ -1,5 +1,5 @@
 # This is just an toy example, because it doesn't make any sense
-# in real world, drafting tokens for the model with similar size.
+# in real world, drafting tokens for the model with smaller size.
 apiVersion: llmaz.io/v1alpha1
 kind: OpenModel
 metadata:
@@ -38,10 +38,3 @@ spec:
     backendName: llamacpp
     args:
       - -fa # use flash attention
-    resources:
-      requests:
-        cpu: 4
-        memory: "8Gi"
-      limits:
-        cpu: 4
-        memory: "8Gi"
diff --git a/test/config/backends/llamacpp.yaml b/test/config/backends/llamacpp.yaml
index dfd81d8e..8e3eb9c5 100644
--- a/test/config/backends/llamacpp.yaml
+++ b/test/config/backends/llamacpp.yaml
@@ -29,21 +29,20 @@ spec:
         limits:
           cpu: 2
           memory: 4Gi
-    # TODO: not supported yet, see https://github.com/InftyAI/llmaz/issues/240.
-    # - name: speculative-decoding
-    #   args:
-    #     - -m
-    #     - "{{ .ModelPath }}"
-    #     - -md
-    #     - "{{ .DraftModelPath }}"
-    #     - --host
-    #     - "0.0.0.0"
-    #     - --port
-    #     - "8080"
-    #     - --draft-max
-    #     - "16"
-    #     - --draft-min
-    #     - "5"
+    - name: speculative-decoding
+      args:
+        - -m
+        - "{{ .ModelPath }}"
+        - -md
+        - "{{ .DraftModelPath }}"
+        - --host
+        - "0.0.0.0"
+        - --port
+        - "8080"
+        - --draft-max
+        - "16"
+        - --draft-min
+        - "5"
   startupProbe:
     periodSeconds: 10
     failureThreshold: 30
diff --git a/test/e2e/playground_test.go b/test/e2e/playground_test.go
index 9e73f434..d82bd708 100644
--- a/test/e2e/playground_test.go
+++ b/test/e2e/playground_test.go
@@ -142,32 +142,29 @@ var _ = ginkgo.Describe("playground e2e tests", func() {
 		hpa := &autoscalingv2.HorizontalPodAutoscaler{}
 		gomega.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: playground.Name, Namespace: playground.Namespace}, hpa)).To(gomega.Succeed())
 	})
-	// TODO: add e2e tests.
-	// ginkgo.It("SpeculativeDecoding with llama.cpp", func() {
-	// 	targetModel := wrapper.MakeModel("llama2-7b-q8-gguf").FamilyName("llama2").ModelSourceWithModelHub("Huggingface").ModelSourceWithModelID("TheBloke/Llama-2-7B-GGUF", "llama-2-7b.Q8_0.gguf", "", nil, nil).Obj()
-	// 	gomega.Expect(k8sClient.Create(ctx, targetModel)).To(gomega.Succeed())
-	// 	defer func() {
-	// 		gomega.Expect(k8sClient.Delete(ctx, targetModel)).To(gomega.Succeed())
-	// 	}()
-	// 	draftModel := wrapper.MakeModel("llama2-7b-q2-k-gguf").FamilyName("llama2").ModelSourceWithModelHub("Huggingface").ModelSourceWithModelID("TheBloke/Llama-2-7B-GGUF", "llama-2-7b.Q2_K.gguf", "", nil, nil).Obj()
-	// 	gomega.Expect(k8sClient.Create(ctx, draftModel)).To(gomega.Succeed())
-	// 	defer func() {
-	// 		gomega.Expect(k8sClient.Delete(ctx, draftModel)).To(gomega.Succeed())
-	// 	}()
-
-	// 	playground := wrapper.MakePlayground("llamacpp-speculator", ns.Name).
-	// 		MultiModelsClaim([]string{"llama2-7b-q8-gguf", "llama2-7b-q2-k-gguf"}, coreapi.SpeculativeDecoding).
-	// 		BackendRuntime("llamacpp").BackendLimit("cpu", "4").BackendRequest("memory", "8Gi").
-	// 		Replicas(1).
-	// 		Obj()
-	// 	gomega.Expect(k8sClient.Create(ctx, playground)).To(gomega.Succeed())
-	// 	validation.ValidatePlayground(ctx, k8sClient, playground)
-	// 	validation.ValidatePlaygroundStatusEqualTo(ctx, k8sClient, playground, inferenceapi.PlaygroundAvailable, "PlaygroundReady", metav1.ConditionTrue)
-
-	// 	service := &inferenceapi.Service{}
-	// 	gomega.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: playground.Name, Namespace: playground.Namespace}, service)).To(gomega.Succeed())
-	// 	validation.ValidateService(ctx, k8sClient, service)
-	// 	validation.ValidateServiceStatusEqualTo(ctx, k8sClient, service, inferenceapi.ServiceAvailable, "ServiceReady", metav1.ConditionTrue)
-	// 	validation.ValidateServicePods(ctx, k8sClient, service)
-	// })
+	ginkgo.It("SpeculativeDecoding with llama.cpp", func() {
+		targetModel := wrapper.MakeModel("llama2-7b-q8-gguf").FamilyName("llama2").ModelSourceWithModelHub("Huggingface").ModelSourceWithModelID("TheBloke/Llama-2-7B-GGUF", "llama-2-7b.Q8_0.gguf", "", nil, nil).Obj()
+		gomega.Expect(k8sClient.Create(ctx, targetModel)).To(gomega.Succeed())
+		defer func() {
+			gomega.Expect(k8sClient.Delete(ctx, targetModel)).To(gomega.Succeed())
+		}()
+		draftModel := wrapper.MakeModel("llama2-7b-q2-k-gguf").FamilyName("llama2").ModelSourceWithModelHub("Huggingface").ModelSourceWithModelID("TheBloke/Llama-2-7B-GGUF", "llama-2-7b.Q2_K.gguf", "", nil, nil).Obj()
+		gomega.Expect(k8sClient.Create(ctx, draftModel)).To(gomega.Succeed())
+		defer func() {
+			gomega.Expect(k8sClient.Delete(ctx, draftModel)).To(gomega.Succeed())
+		}()
+
+		playground := wrapper.MakePlayground("llamacpp-speculator", ns.Name).
+			ModelClaims([]string{"llama2-7b-q8-gguf", "llama2-7b-q2-k-gguf"}, []string{"main", "draft"}).
+			BackendRuntime("llamacpp").Replicas(1).Obj()
+		gomega.Expect(k8sClient.Create(ctx, playground)).To(gomega.Succeed())
+		validation.ValidatePlayground(ctx, k8sClient, playground)
+		validation.ValidatePlaygroundStatusEqualTo(ctx, k8sClient, playground, inferenceapi.PlaygroundAvailable, "PlaygroundReady", metav1.ConditionTrue)
+
+		service := &inferenceapi.Service{}
+		gomega.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: playground.Name, Namespace: playground.Namespace}, service)).To(gomega.Succeed())
+		validation.ValidateService(ctx, k8sClient, service)
+		validation.ValidateServiceStatusEqualTo(ctx, k8sClient, service, inferenceapi.ServiceAvailable, "ServiceReady", metav1.ConditionTrue)
+		validation.ValidateServicePods(ctx, k8sClient, service)
+	})
 })


Reviews:
user: login: cr7258
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T14:05:43Z
user: login: kerthcet
body: Only one comment.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T12:52:31Z
user: login: cr7258
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T14:02:45Z
user: login: kerthcet
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T15:04:55Z

Review comments:
diff_hunk: @@ -42,7 +42,6 @@ type BackendRuntimeConfig struct {
 	// ConfigName represents the recommended configuration name for the backend,
 	// It will be inferred from the models in the runtime if not specified, e.g. default,
 	// speculative-decoding.
-	// +kubebuilder:default=default
path: api/inference/v1alpha1/config_types.go
user: login: cr7258
body: > Why remove the default value of the ConfigName field?

llmaz [infers the recommended configuration name](https://github.com/InftyAI/llmaz/blob/v0.1.3/pkg/controller_helper/helper.go#L36) for the backend if not specified. However, the `kubebuilder:default=default` annotation prevents this inference by always setting ConfigName to "default" instead of leaving it as nil, bypassing the role-based detection logic by mistake.
created_at: 2025-05-06T14:05:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,7 +42,6 @@ type BackendRuntimeConfig struct {
 	// ConfigName represents the recommended configuration name for the backend,
 	// It will be inferred from the models in the runtime if not specified, e.g. default,
 	// speculative-decoding.
-	// +kubebuilder:default=default
path: api/inference/v1alpha1/config_types.go
user: login: kerthcet
body: Set the default value here doesn't make any difference, right? The inference is just a guardrail I believe.
created_at: 2025-05-08T12:50:30Z
author_association: MEMBER
diff_hunk: @@ -42,7 +42,6 @@ type BackendRuntimeConfig struct {
 	// ConfigName represents the recommended configuration name for the backend,
 	// It will be inferred from the models in the runtime if not specified, e.g. default,
 	// speculative-decoding.
-	// +kubebuilder:default=default
path: api/inference/v1alpha1/config_types.go
user: login: cr7258
body: If we set the default value here, when we don't define `configName` in the Playground, even though we define main and draft models, the configName will always be set as `default` instead of `speculative-decoding`.
created_at: 2025-05-08T14:02:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,7 +42,6 @@ type BackendRuntimeConfig struct {
 	// ConfigName represents the recommended configuration name for the backend,
 	// It will be inferred from the models in the runtime if not specified, e.g. default,
 	// speculative-decoding.
-	// +kubebuilder:default=default
path: api/inference/v1alpha1/config_types.go
user: login: kerthcet
body: Recalled why we remove this before. Make sense to me.
created_at: 2025-05-08T15:04:55Z
author_association: MEMBER

PR comments:
user: login: cr7258
created_at: 2025-05-06T14:06:36Z
body: /kind feature
author_association: CONTRIBUTOR
user: login: kerthcet
created_at: 2025-05-07T11:30:23Z
body: I will take a look tonight or tomorrow at the latest.

author_association: MEMBER
user: login: kerthcet
created_at: 2025-05-08T15:05:06Z
body: /lgtm
/approve

Thanks @cr7258 
author_association: MEMBER
user: login: kerthcet
created_at: 2025-05-08T15:07:28Z
body: /triage accepted
author_association: MEMBER