Repository: instructlab/instructlab

Issue:
Update llama_cpp_python from 0.2.79 to 0.3 for executing new LLM architectures (granite 3.0)
**Is your feature request related to a problem? Please describe.**
The current deployed version of instructlab requires llama_cpp version 0.2.79, and I want to run the new IBM Granite architecture, which is available in the latest versions of llama_cpp.

**Describe the solution you'd like**
A package increment, or change version intervals to make it easier for forks and open source projects to add and test newer libraries (i.e. `llama_cpp_python>=0.2.79<=0.3`):

https://github.com/instructlab/instructlab/blob/f15a4166555a4b9c0f93c2e0e1cfecd638671578/requirements.txt#L14

**Additional context**

1. Trying to execute granite 3.0 in the current llama_cpp version (0.2.79):

```python
>>> import llama_cpp
>>> llama = llama_cpp.Llama('models/llm/granite-3.0-8b-instruct/granite-3.0-8B-instruct-F16_llama_cpp.gguf')
llama_model_loader: loaded meta data with 36 key-value pairs and 362 tensors from models/llm/granite-3.0-8b-instruct/granite-3.0-8B-instruct-F16_llama_cpp.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = granite
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Granite 3.0 8b Instruct
llama_model_loader: - kv   3:                           general.finetune str              = instruct
llama_model_loader: - kv   4:                           general.basename str              = granite-3.0
llama_model_loader: - kv   5:                         general.size_label str              = 8B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                               general.tags arr[str,3]       = ["language", "granite-3.0", "text-gen...
llama_model_loader: - kv   8:                        granite.block_count u32              = 40
llama_model_loader: - kv   9:                     granite.context_length u32              = 4096
llama_model_loader: - kv  10:                   granite.embedding_length u32              = 4096
llama_model_loader: - kv  11:                granite.feed_forward_length u32              = 12800
llama_model_loader: - kv  12:               granite.attention.head_count u32              = 32
llama_model_loader: - kv  13:            granite.attention.head_count_kv u32              = 8
llama_model_loader: - kv  14:                     granite.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  15:   granite.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  16:                          general.file_type u32              = 1
llama_model_loader: - kv  17:                         granite.vocab_size u32              = 49155
llama_model_loader: - kv  18:               granite.rope.dimension_count u32              = 128
llama_model_loader: - kv  19:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  20:                    granite.attention.scale f32              = 0.007812
llama_model_loader: - kv  21:                    granite.embedding_scale f32              = 12.000000
llama_model_loader: - kv  22:                     granite.residual_scale f32              = 0.220000
llama_model_loader: - kv  23:                        granite.logit_scale f32              = 16.000000
llama_model_loader: - kv  24:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  25:                         tokenizer.ggml.pre str              = refact
llama_model_loader: - kv  26:                      tokenizer.ggml.tokens arr[str,49155]   = ["
", "<fim_prefix>", "...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,49155]   = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  28:                      tokenizer.ggml.merges arr[str,48891]   = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ...
llama_model_loader: - kv  29:                tokenizer.ggml.bos_token_id u32              = 0
llama_model_loader: - kv  30:                tokenizer.ggml.eos_token_id u32              = 0
llama_model_loader: - kv  31:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  32:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  33:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|start_of_r...
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   81 tensors
llama_model_loader: - type  f16:  281 tensors
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'granite'
llama_load_model_from_file: failed to load model
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/lib/python3.12/site-packages/llama_cpp/llama.py", line 358, in __init__
    self._model = self._stack.enter_context(contextlib.closing(_LlamaModel(
                                                               ^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.12/site-packages/llama_cpp/_internals.py", line 54, in __init__
    raise ValueError(f"Failed to load model from file: {path_model}")
```

2. Triying to execute granite in the newest llama_cpp version (0.3.1):

```python
>>> llama = llama_cpp.Llama('models/llm/granite-3.0-8b-instruct/granite-3.0-8B-instruct-F16_llama_cpp.gguf')
llama_model_loader: loaded meta data with 36 key-value pairs and 362 tensors from models/llm/granite-3.0-8b-instruct/granite-3.0-8B-instruct-F16_llama_cpp.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = granite
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Granite 3.0 8b Instruct
llama_model_loader: - kv   3:                           general.finetune str              = instruct
llama_model_loader: - kv   4:                           general.basename str              = granite-3.0
llama_model_loader: - kv   5:                         general.size_label str              = 8B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                               general.tags arr[str,3]       = ["language", "granite-3.0", "text-gen...
llama_model_loader: - kv   8:                        granite.block_count u32              = 40
llama_model_loader: - kv   9:                     granite.context_length u32              = 4096
llama_model_loader: - kv  10:                   granite.embedding_length u32              = 4096
llama_model_loader: - kv  11:                granite.feed_forward_length u32              = 12800
llama_model_loader: - kv  12:               granite.attention.head_count u32              = 32
llama_model_loader: - kv  13:            granite.attention.head_count_kv u32              = 8
llama_model_loader: - kv  14:                     granite.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  15:   granite.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  16:                          general.file_type u32              = 1
llama_model_loader: - kv  17:                         granite.vocab_size u32              = 49155
llama_model_loader: - kv  18:               granite.rope.dimension_count u32              = 128
llama_model_loader: - kv  19:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  20:                    granite.attention.scale f32              = 0.007812
llama_model_loader: - kv  21:                    granite.embedding_scale f32              = 12.000000
llama_model_loader: - kv  22:                     granite.residual_scale f32              = 0.220000
llama_model_loader: - kv  23:                        granite.logit_scale f32              = 16.000000
llama_model_loader: - kv  24:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  25:                         tokenizer.ggml.pre str              = refact
llama_model_loader: - kv  26:                      tokenizer.ggml.tokens arr[str,49155]   = ["
", "<fim_prefix>", "...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,49155]   = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  28:                      tokenizer.ggml.merges arr[str,48891]   = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ...
llama_model_loader: - kv  29:                tokenizer.ggml.bos_token_id u32              = 0
llama_model_loader: - kv  30:                tokenizer.ggml.eos_token_id u32              = 0
llama_model_loader: - kv  31:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  32:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  33:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|start_of_r...
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   81 tensors
llama_model_loader: - type  f16:  281 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 22
llm_load_vocab: token to piece cache size = 0.2826 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = granite
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 49155
llm_load_print_meta: n_merges         = 48891
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 40
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 1.6e+01
llm_load_print_meta: n_ff             = 12800
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 3B
llm_load_print_meta: model ftype      = F16
llm_load_print_meta: model params     = 8.17 B
llm_load_print_meta: model size       = 15.22 GiB (16.00 BPW)
llm_load_print_meta: general.name     = Granite 3.0 8b Instruct
llm_load_print_meta: BOS token        = 0 '
'
llm_load_print_meta: EOS token        = 0 '
'
llm_load_print_meta: UNK token        = 0 '
'
llm_load_print_meta: PAD token        = 0 '
'
llm_load_print_meta: LF token         = 145 'Ä'
llm_load_print_meta: EOG token        = 0 '
'
llm_load_print_meta: max token length = 512
llm_load_print_meta: f_embedding_scale = 12.000000
llm_load_print_meta: f_residual_scale  = 0.220000
llm_load_print_meta: f_attention_scale = 0.007812
llm_load_tensors: ggml ctx size =    0.17 MiB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/41 layers to GPU
llm_load_tensors:        CPU buffer size = 15585.29 MiB
..................................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:        CPU KV buffer size =    80.00 MiB
llama_new_context_with_model: KV self size  =   80.00 MiB, K (f16):   40.00 MiB, V (f16):   40.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.19 MiB
llama_new_context_with_model:        CPU compute buffer size =   104.01 MiB
llama_new_context_with_model: graph nodes  = 1368
llama_new_context_with_model: graph splits = 643
AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 1 | LLAMAFILE = 1 |
Model metadata: {'tokenizer.chat_template': "{%- if tools %}\n    {{- '
available_tools
\n' }}\n    {%- for tool in tools %}\n    {{- tool | tojson(indent=4) }}\n    {%- if not loop.last %}\n        {{- '\n\n' }}\n    {%- endif %}\n    {%- endfor %}\n    {{- '
\n' }}\n{%- endif %}\n{%- for message in messages %}\n    {%- if message['role'] == 'system' %}\n    {{- '
system
' + message['content'] + '
\n' }}\n    {%- elif message['role'] == 'user' %}\n    {{- '
user
' + message['content'] + '
\n' }}\n    {%- elif message['role'] == 'assistant' %}\n    {{- '
assistant
'  + message['content'] + '
\n' }}\n    {%- elif message['role'] == 'assistant_tool_call' %}\n    {{- '
assistant

' + message['content'] + '
\n' }}\n    {%- elif message['role'] == 'tool_response' %}\n    {{- '
tool_response
' + message['content'] + '
\n' }}\n    {%- endif %}\n    {%- if loop.last and add_generation_prompt %}\n    {{- '
assistant
' }}\n    {%- endif %}\n{%- endfor %}", 'tokenizer.ggml.add_bos_token': 'false', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '0', 'tokenizer.ggml.pre': 'refact', 'tokenizer.ggml.model': 'gpt2', 'granite.residual_scale': '0.220000', 'general.architecture': 'granite', 'granite.feed_forward_length': '12800', 'tokenizer.ggml.add_space_prefix': 'false', 'granite.rope.dimension_count': '128', 'granite.rope.freq_base': '10000.000000', 'granite.attention.head_count_kv': '8', 'tokenizer.ggml.padding_token_id': '0', 'granite.attention.head_count': '32', 'granite.embedding_length': '4096', 'granite.vocab_size': '49155', 'granite.context_length': '4096', 'granite.block_count': '40', 'granite.attention.layer_norm_rms_epsilon': '0.000010', 'general.quantization_version': '2', 'general.license': 'apache-2.0', 'granite.attention.scale': '0.007812', 'general.file_type': '1', 'general.finetune': 'instruct', 'granite.embedding_scale': '12.000000', 'general.basename': 'granite-3.0', 'tokenizer.ggml.bos_token_id': '0', 'general.type': 'model', 'granite.logit_scale': '16.000000', 'general.name': 'Granite 3.0 8b Instruct', 'general.size_label': '8B'}
Available chat formats from metadata: chat_template.default
Using gguf chat template: {%- if tools %}
    {{- '
available_tools

' }}
    {%- for tool in tools %}
    {{- tool | tojson(indent=4) }}
    {%- if not loop.last %}
        {{- '

' }}
    {%- endif %}
    {%- endfor %}
    {{- '

' }}
{%- endif %}
{%- for message in messages %}
    {%- if message['role'] == 'system' %}
    {{- '
system
' + message['content'] + '

' }}
    {%- elif message['role'] == 'user' %}
    {{- '
user
' + message['content'] + '

' }}
    {%- elif message['role'] == 'assistant' %}
    {{- '
assistant
'  + message['content'] + '

' }}
    {%- elif message['role'] == 'assistant_tool_call' %}
    {{- '
assistant

' + message['content'] + '

' }}
    {%- elif message['role'] == 'tool_response' %}
    {{- '
tool_response
' + message['content'] + '

' }}
    {%- endif %}
    {%- if loop.last and add_generation_prompt %}
    {{- '
assistant
' }}
    {%- endif %}
{%- endfor %}
Using chat eos_token: 

Using chat bos_token: 

```



Discussion:
There is an existing PR for this: https://github.com/instructlab/instructlab/pull/2368



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f059af5735..5455a5c360 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,8 @@
 
 ### Breaking Changes
 
+* llama-cpp-python has been bumped to 0.3.2. This allows for serving of Granite 3.0 GGUF Models. With this change, some previous handling of context window size has been modified to work with the 0.3.z releases of llama-cpp-python.
+
 ### Features
 
 * A new command `ilab model upload` has been introduced so users can now upload their trained models to [Hugging Face](https://huggingface.co/) via the `ilab` CLI
diff --git a/README.md b/README.md
index 30089971de..7a9f810634 100644
--- a/README.md
+++ b/README.md
@@ -118,7 +118,7 @@ For an overview of the full workflow, see the [workflow diagram](./docs/workflow
 
    ⚠️ The `python3` binary shown in the installation guides are the Python version that you installed in the above step. The command can also be `python3.11` or `python3.10` instead of `python3`. You can check Python's version by `python3 -V`.
 
-   ⏳ `pip install` may take some time, depending on your internet connection. In case the installation fails with error ``unsupported instruction `vpdpbusd'``, append `-C cmake.args="-DLLAMA_NATIVE=off"` to `pip install` command.
+   ⏳ `pip install` may take some time, depending on your internet connection. In case the installation fails with error ``unsupported instruction `vpdpbusd'``, append `-C cmake.args="-DGGML_NATIVE=off"` to `pip install` command.
 
    See [the GPU acceleration documentation](./docs/gpu-acceleration.md) for how to to enable hardware acceleration for interaction and training on AMD ROCm, Apple Metal Performance Shaders (MPS), and Nvidia CUDA.
 
@@ -149,9 +149,9 @@ For an overview of the full workflow, see the [workflow diagram](./docs/workflow
 
    *Additional Build Argument for Intel Macs*
    If you have a Mac with an Intel CPU, you must add a prefix of
-   `CMAKE_ARGS="-DLLAMA_METAL=off"` to the `pip install` command to ensure
+   `CMAKE_ARGS="-DGGML_METAL=off"` to the `pip install` command to ensure
    that the build is done without Apple M-series GPU support.
-   `(venv) $ CMAKE_ARGS="-DLLAMA_METAL=off" pip install ...`
+   `(venv) $ CMAKE_ARGS="-DGGML_METAL=off" pip install ...`
 
 #### Install with AMD ROCm
 
@@ -161,12 +161,12 @@ For an overview of the full workflow, see the [workflow diagram](./docs/workflow
    python3 -m venv --upgrade-deps venv
    source venv/bin/activate
    pip cache remove llama_cpp_python
-   CMAKE_ARGS="-DLLAMA_HIPBLAS=on \
+   CMAKE_ARGS="-DGGML_HIPBLAS=on \
       -DAMDGPU_TARGETS=all \
       -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang \
       -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ \
       -DCMAKE_PREFIX_PATH=/opt/rocm \
-      -DLLAMA_NATIVE=off" \
+      -DGGML_NATIVE=off" \
       pip install 'instructlab[rocm]' \
       --extra-index-url https://download.pytorch.org/whl/rocm6.0
    ```
@@ -181,7 +181,7 @@ For an overview of the full workflow, see the [workflow diagram](./docs/workflow
    python3 -m venv --upgrade-deps venv
    source venv/bin/activate
    pip cache remove llama_cpp_python
-   CMAKE_ARGS="-DLLAMA_CUDA=on -DLLAMA_NATIVE=off" pip install 'instructlab[cuda]'
+   CMAKE_ARGS="-DGGML_CUDA=on -DGGML_NATIVE=off" pip install 'instructlab[cuda]'
    pip install vllm@git+https://github.com/opendatahub-io/vllm@v0.6.2
    ```
 
diff --git a/containers/cuda/Containerfile b/containers/cuda/Containerfile
index 19e91c9384..c1390045b8 100644
--- a/containers/cuda/Containerfile
+++ b/containers/cuda/Containerfile
@@ -185,7 +185,7 @@ RUN source ${VIRTUAL_ENV}/bin/activate \
     fi \
     && pip install torch psutil \
     && pip install flash_attn --no-build-isolation \
-    && pip install "${INSTRUCTLAB_PKG}" -C cmake.args="-DLLAMA_CUDA=on" -C cmake.args="-DLLAMA_NATIVE=off" \
+    && pip install "${INSTRUCTLAB_PKG}" -C cmake.args="-DGGML_CUDA=on" -C cmake.args="-DGGML_NATIVE=off" \
     && pip install vllm@git+https://github.com/opendatahub-io/vllm@v0.6.2
 
 ENV TORCH_HOME="${VIRTUAL_ENV}/lib64/${PYTHON}/site-packages/torch"
diff --git a/containers/hpu/Containerfile b/containers/hpu/Containerfile
index 4ede157509..c4de6d8777 100644
--- a/containers/hpu/Containerfile
+++ b/containers/hpu/Containerfile
@@ -26,7 +26,7 @@ COPY containers/sitecustomize.py ${VIRTUAL_ENV}/lib/${PYTHON}/site-packages/
 COPY containers/bin/debug-* ${VIRTUAL_ENV}/bin/
 
 COPY . /tmp/instructlab
-RUN CMAKE_ARGS="-DLLAMA_NATIVE=off" \
+RUN CMAKE_ARGS="-DGGML_NATIVE=off" \
         ${VIRTUAL_ENV}/bin/pip install "/tmp/instructlab[hpu]" && \
     find ${VIRTUAL_ENV} -name __pycache__ | xargs rm -rf
 
diff --git a/containers/rocm/Containerfile b/containers/rocm/Containerfile
index a6ca025953..9a6ecab124 100644
--- a/containers/rocm/Containerfile
+++ b/containers/rocm/Containerfile
@@ -65,7 +65,7 @@ FROM pytorch AS llama
 RUN --mount=type=cache,sharing=locked,id=pipcache,target=${PIP_CACHE_DIR},mode=777,z \
     pip cache remove llama_cpp_python && \
     umask 0000 && \
-    CMAKE_ARGS="-DLLAMA_HIPBLAS=on -DCMAKE_C_COMPILER=/usr/bin/clang-${CLANG_VER} -DCMAKE_CXX_COMPILER=/usr/bin/clang++-${CLANG_VER} -DAMDGPU_TARGETS=${AMDGPU_ARCH}" \
+    CMAKE_ARGS="-DGGML_HIPBLAS=on -DCMAKE_C_COMPILER=/usr/bin/clang-${CLANG_VER} -DCMAKE_CXX_COMPILER=/usr/bin/clang++-${CLANG_VER} -DAMDGPU_TARGETS=${AMDGPU_ARCH}" \
     FORCE_CMAKE=1 \
     ${VIRTUAL_ENV}/bin/pip install -c ${VIRTUAL_ENV}/constraints.txt llama-cpp-python
 
diff --git a/docs/gpu-acceleration.md b/docs/gpu-acceleration.md
index b9edfc0f91..283b9120c6 100644
--- a/docs/gpu-acceleration.md
+++ b/docs/gpu-acceleration.md
@@ -126,7 +126,7 @@ export PATH=$PATH:$CUDA_HOME/bin
 
 # Recompile llama-cpp-python using CUDA
 pip cache remove llama_cpp_python
-pip install --force-reinstall --no-deps llama_cpp_python==0.2.79 -C cmake.args="-DLLAMA_CUDA=on"
+pip install --force-reinstall --no-deps llama_cpp_python==0.2.79 -C cmake.args="-DGGML_CUDA=on"
 
 # Re-install InstructLab
 pip install ./instructlab[cuda]
@@ -139,7 +139,7 @@ supports GCC v14.1+.
 ```shell
 # Recompile llama-cpp-python using CUDA
 sudo dnf install clang17
-CUDAHOSTCXX=$(which clang++-17) pip install --force-reinstall llama_cpp_python==0.2.79 -C cmake.args="-DLLAMA_CUDA=on"
+CUDAHOSTCXX=$(which clang++-17) pip install --force-reinstall llama_cpp_python==0.2.79 -C cmake.args="-DGGML_CUDA=on"
 ```
 
 Proceed to the `Initialize` section of
@@ -221,14 +221,14 @@ we'll include that in our build command as follows:
 ```shell
 export PATH=/opt/rocm/llvm/bin:$PATH
 pip cache remove llama_cpp_python
-CMAKE_ARGS="-DLLAMA_HIPBLAS=on -DCMAKE_C_COMPILER='/opt/rocm/llvm/bin/clang' -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ -DCMAKE_PREFIX_PATH=/opt/rocm -DAMDGPU_TARGETS=gfx1100" FORCE_CMAKE=1 pip install --force-reinstall llama_cpp_python==0.2.79
+CMAKE_ARGS="-DGGML_HIPBLAS=on -DCMAKE_C_COMPILER='/opt/rocm/llvm/bin/clang' -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ -DCMAKE_PREFIX_PATH=/opt/rocm -DAMDGPU_TARGETS=gfx1100" FORCE_CMAKE=1 pip install --force-reinstall llama_cpp_python==0.2.79
 ```
 
 > **Note:** This is explicitly forcing the build to use the ROCm compilers and
 > prefix path for dependency resolution in the CMake build.  This works around
 > an issue in the CMake and ROCm version in Fedora 39 and below and is fixed in
 > Fedora 40.  With Fedora 40's ROCm packages, use
-> `CMAKE_ARGS="-DLLAMA_HIPBLAS=on -DCMAKE_C_COMPILER=/usr/bin/clang
+> `CMAKE_ARGS="-DGGML_HIPBLAS=on -DCMAKE_C_COMPILER=/usr/bin/clang
 > -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DAMDGPU_TARGETS=gfx1100"` instead.
 
 Once that package is installed, recompile `ilab` with `pip install .[rocm]`.  You also
@@ -269,7 +269,7 @@ add a few options to ensure it gets installed over the existing package:
 
 ```shell
 pip cache remove llama_cpp_python
-pip install --force-reinstall llama_cpp_python==0.2.79 -C cmake.args="-DLLAMA_METAL=on"
+pip install --force-reinstall llama_cpp_python==0.2.79 -C cmake.args="-DGGML_METAL=on"
 ```
 
 Once that package is installed, recompile `ilab` with `pip install .[mps]` and skip
diff --git a/requirements.txt b/requirements.txt
index b6b7130687..ee3295f837 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -11,7 +11,7 @@ instructlab-quantize>=0.1.0
 instructlab-schema>=0.4.0
 instructlab-sdg>=0.6.2
 instructlab-training>=0.6.0
-llama_cpp_python[server]==0.2.79
+llama_cpp_python[server]==0.3.2
 mlx>=0.5.1,<0.6.0; sys_platform == 'darwin' and platform_machine == 'arm64'
 numpy>=1.26.4,<2.0.0
 openai>=1.13.3
diff --git a/scripts/infra/cloud-instance.sh b/scripts/infra/cloud-instance.sh
index 88a5f9c963..0fbd0d293f 100755
--- a/scripts/infra/cloud-instance.sh
+++ b/scripts/infra/cloud-instance.sh
@@ -300,7 +300,7 @@ pip_install_with_nvidia() {
 pip_install_with_amd() {
     "${BASH_SOURCE[0]}" "$CLOUD_TYPE" ssh -n "$INSTANCE_NAME" "pushd instructlab && source venv/bin/activate && pip cache remove llama_cpp_python && \
     pip install -e .[rocm] --extra-index-url https://download.pytorch.org/whl/rocm6.0 \
-   -C cmake.args='-DLLAMA_HIPBLAS=on' \
+   -C cmake.args='-DGGML_HIPBLAS=on' \
    -C cmake.args='-DAMDGPU_TARGETS=all' \
    -C cmake.args='-DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang' \
    -C cmake.args='-DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++' \
diff --git a/src/instructlab/cli/model/serve.py b/src/instructlab/cli/model/serve.py
index ad0f266947..8fe6a0ef0e 100644
--- a/src/instructlab/cli/model/serve.py
+++ b/src/instructlab/cli/model/serve.py
@@ -10,6 +10,8 @@
 
 # First Party
 from instructlab import clickext
+from instructlab.configuration import write_config
+from instructlab.defaults import DEFAULTS
 from instructlab.model.backends import backends
 from instructlab.model.serve_backend import serve_backend, signal_handler
 
@@ -117,6 +119,14 @@ def serve(
 
     warn_for_unsupported_backend_param(ctx)
 
+    # we need to keep track of the max_ctx_size being uses in the active server so we don't overflow the context
+    if max_ctx_size != DEFAULTS.MAX_CONTEXT_SIZE:
+        logger.info(
+            f"Setting current_max_ctx_size in the serve config to {max_ctx_size}"
+        )
+        ctx.obj.config.serve.server.current_max_ctx_size = max_ctx_size
+        write_config(ctx.obj.config)
+
     serve_backend(
         ctx,
         model_path,
diff --git a/src/instructlab/configuration.py b/src/instructlab/configuration.py
index f265be636f..eb5c60d436 100644
--- a/src/instructlab/configuration.py
+++ b/src/instructlab/configuration.py
@@ -196,7 +196,7 @@ class _serve_llama_cpp(BaseModel):
         description="Number of model layers to offload to GPU. -1 means all layers.",
     )
     max_ctx_size: PositiveInt = Field(
-        default=4096,
+        default=DEFAULTS.MAX_CONTEXT_SIZE,
         description="Maximum number of tokens that can be processed by the model.",
     )
     llm_family: str = Field(
@@ -211,6 +211,15 @@ class _serve_server(BaseModel):
 
     host: StrictStr = Field(default="127.0.0.1", description="Host to serve on.")
     port: StrictInt = Field(default=8000, description="Port to serve on.")
+    backend_type: str = Field(
+        default="",
+        description="Backend Instance Type",
+        examples=["llama-cpp", "vllm"],
+    )
+    current_max_ctx_size: PositiveInt = Field(
+        default=DEFAULTS.MAX_CONTEXT_SIZE,
+        description="Maximum number of tokens that can be processed by the currently served model.",
+    )
 
 
 class _serve(BaseModel):
@@ -947,9 +956,13 @@ def read_config(
     config_file: str | os.PathLike[str] | None = None,
 ) -> Config:
     """Reads configuration from disk."""
+    # Third Party
+    from filelock import FileLock
+
+    lock = FileLock(DEFAULTS.CONFIG_FILE_LOCK, timeout=1)
     config_file = DEFAULTS.CONFIG_FILE if config_file is None else config_file
     try:
-        with open(config_file, "r", encoding="utf-8") as yamlfile:
+        with lock, open(config_file, "r", encoding="utf-8") as yamlfile:
             content = yaml.load(yamlfile)
             if not isinstance(content, dict):
                 raise ConfigException(
@@ -1147,8 +1160,12 @@ def write_config_to_yaml(cfg: Config, file_path: str):
         cfg (Config): The Pydantic model to write to YAML.
         file_path (str): The path to the YAML file to write.
     """
+    # Third Party
+    from filelock import FileLock
+
+    lock = FileLock(DEFAULTS.CONFIG_FILE_LOCK)
     commented_map = config_to_commented_map(cfg)
-    with open(file_path, "w", encoding="utf-8") as yaml_file:
+    with lock, open(file_path, "w", encoding="utf-8") as yaml_file:
         yaml.dump(commented_map, yaml_file)
 
 
diff --git a/src/instructlab/defaults.py b/src/instructlab/defaults.py
index 4fde1f64c4..a4be16f390 100644
--- a/src/instructlab/defaults.py
+++ b/src/instructlab/defaults.py
@@ -8,6 +8,7 @@
 
 ILAB_PACKAGE_NAME = "instructlab"
 CONFIG_FILENAME = "config.yaml"
+CONFIG_LOCK_FILENAME = "config.yaml.lock"
 CONFIG_VERSION = "1.0.0"
 
 # Model families understood by ilab
@@ -128,6 +129,10 @@ def DATASETS_DIR(self) -> str:
     def CONFIG_FILE(self) -> str:
         return path.join(self._config_dir, CONFIG_FILENAME)
 
+    @property
+    def CONFIG_FILE_LOCK(self) -> str:
+        return path.join(self._config_dir, CONFIG_LOCK_FILENAME)
+
     @property
     def MODELS_DIR(self) -> str:
         return path.join(self._cache_home, STORAGE_DIR_NAMES.MODELS)
diff --git a/src/instructlab/model/chat.py b/src/instructlab/model/chat.py
index 0eed4f328e..85c029a431 100644
--- a/src/instructlab/model/chat.py
+++ b/src/instructlab/model/chat.py
@@ -34,6 +34,7 @@
 # Local
 from ..client_utils import http_client
 from ..utils import get_cli_helper_sysprompt, get_model_arch, get_sysprompt
+from .backends import backends
 
 logger = logging.getLogger(__name__)
 
@@ -197,6 +198,8 @@ def chat(
     # First Party
     from instructlab.model.backends.common import is_temp_server_running
 
+    max_ctx_size = None
+    backend_type = ctx.obj.config.serve.server.backend_type
     serve_server = ctx.obj.config.serve.server
     users_endpoint_url = cfg.get_api_base(serve_server.host, serve_server.port)
 
@@ -220,10 +223,9 @@ def chat(
             logger.warning(
                 "Setting serving log file (--serving-log-file) is not supported when the server is already running"
             )
-    else:
-        # First Party
-        from instructlab.model.backends import backends
+        max_ctx_size = ctx.obj.config.serve.server.current_max_ctx_size
 
+    else:
         # If a log file is specified, write logs to the file
         root_logger = logging.getLogger()
         if serving_log_file:
@@ -235,6 +237,11 @@ def chat(
             model_path=model,
             log_file=serving_log_file,
         )
+        backend_type = backend_instance.get_backend_type()
+        # if max tokens is the default, read from the serve cfg in case that has been overriden
+        # if serve has a different max_ctx_size, that is what we use.
+        if max_ctx_size is None and backend_type == backends.LLAMA_CPP:
+            max_ctx_size = ctx.obj.config.serve.llama_cpp.max_ctx_size
         try:
             # Run the llama server
             api_base = backend_instance.run_detached(http_client(ctx.params))
@@ -275,7 +282,6 @@ def chat(
 
                 # Currently, we only present a single model so we can safely assume that the first model
                 server_model = models.data[0].id if models is not None else None
-
                 # override 'model' with the first returned model if not provided so that the chat print
                 # the model used by the server
                 model = (
@@ -308,7 +314,9 @@ def chat(
             session=session,
             qq=quick_question,
             max_tokens=max_tokens,
+            max_ctx_size=max_ctx_size,
             temperature=temperature,
+            backend_type=backend_type,
         )
     except ChatException as exc:
         click.secho(f"Executing chat failed with: {exc}", fg="red")
@@ -338,7 +346,9 @@ def __init__(
         loaded=None,
         log_file=None,
         max_tokens=None,
+        max_ctx_size=None,
         temperature=1.0,
+        backend_type="",
     ):
         self.client = client
         self.model = model
@@ -347,7 +357,9 @@ def __init__(
         self.loaded = loaded
         self.log_file = log_file
         self.max_tokens = max_tokens
+        self.max_ctx_size = max_ctx_size
         self.temperature = temperature
+        self.backend_type = backend_type
 
         self.console = Console()
 
@@ -656,6 +668,27 @@ def start_prompt(
         try:
             while True:
                 # Loop to catch situations where we need to retry, such as context length exceeded
+                # We need to catch these errors before they hit the server or else it will crash.
+                # as of llama_cpp_python 0.3.z, BadRequestErrors cause the server to become unavailable
+                if self.backend_type == backends.LLAMA_CPP:
+                    total_context_size = 0
+                    # this handling can apply to llama-cpp-python only. The exception handling below should still exist for vLLM.
+                    while True:
+                        # we need to loop this. If we only remove 1 message, but it isn't enough to fit our new message in, we can hit the error
+                        # if you have 3 messages in the list, and the last one is 127 tokens with a 128 context window, you need to drop the first two in order to fit the third
+                        for msg in self.info["messages"]:
+                            total_context_size += len(msg["content"])
+                        if (
+                            self.max_ctx_size is not None
+                            and self.max_ctx_size < total_context_size
+                            and len(self.info["messages"]) > 1
+                        ):
+                            self.info["messages"] = self.info["messages"][1:]
+                            logger.debug(
+                                "Message too large for context size. Dropping from queue."
+                            )
+                        else:
+                            break
                 try:
                     response = self.client.chat.completions.create(
                         model=self.model,
@@ -664,6 +697,7 @@ def start_prompt(
                         **create_params,
                     )
                 except openai.BadRequestError as e:
+                    # we still want to handle this the same for vLLM
                     logger.debug(f"BadRequestError: {e}")
                     if e.code == "context_length_exceeded":
                         if len(self.info["messages"]) > 1:
@@ -765,7 +799,9 @@ def chat_cli(
     session,
     qq,
     max_tokens,
+    max_ctx_size,
     temperature,
+    backend_type,
 ):
     """Starts a CLI-based chat with the server"""
     client = OpenAI(
@@ -832,6 +868,8 @@ def chat_cli(
         loaded=loaded,
         temperature=(temperature if temperature is not None else config.temperature),
         max_tokens=(max_tokens if max_tokens else config.max_tokens),
+        max_ctx_size=max_ctx_size,
+        backend_type=backend_type,
     )
 
     if not qq and session is None:
diff --git a/src/instructlab/model/serve_backend.py b/src/instructlab/model/serve_backend.py
index b311585577..c133742068 100644
--- a/src/instructlab/model/serve_backend.py
+++ b/src/instructlab/model/serve_backend.py
@@ -7,6 +7,7 @@
 
 # First Party
 from instructlab import log
+from instructlab.configuration import write_config
 from instructlab.model.backends import backends
 from instructlab.model.backends.common import ServerException
 from instructlab.model.backends.server import BackendServer
@@ -50,6 +51,9 @@ def serve_backend(
     except (ValueError, AttributeError) as e:
         raise ValueError(f"Failed to determine backend: {e}") from e
 
+    logger.info(f"Setting current_max_ctx_size in the serve config to {max_ctx_size}")
+    ctx.obj.config.serve.server.backend_type = backend
+    write_config(ctx.obj.config)
     if chat_template is None:
         chat_template = ctx.obj.config.serve.chat_template
 
diff --git a/src/instructlab/taxonomy/diff.py b/src/instructlab/taxonomy/diff.py
index 92daeca8cc..465ccf3ea7 100644
--- a/src/instructlab/taxonomy/diff.py
+++ b/src/instructlab/taxonomy/diff.py
@@ -77,6 +77,8 @@ def diff(ctx, taxonomy_path, taxonomy_base, yaml_rules, quiet):
                     )
                     raise SystemExit(1) from exc
             for f in taxonomy_files:
+                if f is None:
+                    continue
                 click.echo(f)
 
     # validate new or changed taxonomy files
diff --git a/tox.ini b/tox.ini
index 5cff770a4d..7848fe49f4 100644
--- a/tox.ini
+++ b/tox.ini
@@ -14,7 +14,7 @@ passenv =
 # are huge. This reduces venv from 5.7 GB to 1.5 GB.
 setenv =
     PIP_EXTRA_INDEX_URL=https://download.pytorch.org/whl/cpu
-    CMAKE_ARGS={env:CMAKE_ARGS:-DLLAMA_NATIVE=off}
+    CMAKE_ARGS={env:CMAKE_ARGS:-DGGML_NATIVE=off}
     ILAB_MAX_STABLE_VRAM_WAIT=0
 package = wheel
 wheel_build_env = pkg


Test patch:
diff --git a/.github/workflows/e2e-nvidia-l4-x1.yml b/.github/workflows/e2e-nvidia-l4-x1.yml
index 716ff1621f..fcf41686d4 100644
--- a/.github/workflows/e2e-nvidia-l4-x1.yml
+++ b/.github/workflows/e2e-nvidia-l4-x1.yml
@@ -123,7 +123,7 @@ jobs:
           nvidia-smi
           python3.11 -m pip cache remove llama_cpp_python
 
-          CMAKE_ARGS="-DLLAMA_CUDA=on" python3.11 -m pip install -v .
+          CMAKE_ARGS="-DGGML_CUDA=on" python3.11 -m pip install -v .
 
           # https://github.com/instructlab/instructlab/issues/1821
           # install with Torch and build dependencies installed
diff --git a/.github/workflows/e2e-nvidia-l40s-x4.yml b/.github/workflows/e2e-nvidia-l40s-x4.yml
index 656c6acc25..5dca4b4dbe 100644
--- a/.github/workflows/e2e-nvidia-l40s-x4.yml
+++ b/.github/workflows/e2e-nvidia-l40s-x4.yml
@@ -131,7 +131,7 @@ jobs:
           nvidia-smi
           python3.11 -m pip cache remove llama_cpp_python
 
-          CMAKE_ARGS="-DLLAMA_CUDA=on" python3.11 -m pip install -v .
+          CMAKE_ARGS="-DGGML_CUDA=on" python3.11 -m pip install -v .
 
           # https://github.com/instructlab/instructlab/issues/1821
           # install with Torch and build dependencies installed
diff --git a/.github/workflows/e2e-nvidia-l40s-x8.yml b/.github/workflows/e2e-nvidia-l40s-x8.yml
index 6721a85f11..9920d7b3ad 100644
--- a/.github/workflows/e2e-nvidia-l40s-x8.yml
+++ b/.github/workflows/e2e-nvidia-l40s-x8.yml
@@ -131,7 +131,7 @@ jobs:
           nvidia-smi
           python3.11 -m pip cache remove llama_cpp_python
 
-          CMAKE_ARGS="-DLLAMA_CUDA=on" python3.11 -m pip install -v .
+          CMAKE_ARGS="-DGGML_CUDA=on" python3.11 -m pip install -v .
 
           # https://github.com/instructlab/instructlab/issues/1821
           # install with Torch and build dependencies installed
diff --git a/.github/workflows/e2e-nvidia-t4-x1.yml b/.github/workflows/e2e-nvidia-t4-x1.yml
index f2cda6d26b..793ea3fdd8 100644
--- a/.github/workflows/e2e-nvidia-t4-x1.yml
+++ b/.github/workflows/e2e-nvidia-t4-x1.yml
@@ -121,7 +121,7 @@ jobs:
           nvidia-smi
           python3.11 -m pip cache remove llama_cpp_python
 
-          CMAKE_ARGS="-DLLAMA_CUDA=on" python3.11 -m pip install -v .
+          CMAKE_ARGS="-DGGML_CUDA=on" python3.11 -m pip install -v .
 
           # https://github.com/instructlab/instructlab/issues/1821
           # install with Torch and build dependencies installed
diff --git a/scripts/functional-tests.sh b/scripts/functional-tests.sh
index d867ec2dd7..ad101ac7aa 100755
--- a/scripts/functional-tests.sh
+++ b/scripts/functional-tests.sh
@@ -250,18 +250,7 @@ test_ctx_size(){
     How many tokens could you take today. Could you tell me about the time you could only take twenty five tokens" &> "$TEST_CTX_SIZE_LAB_CHAT_LOG_FILE" &
     PID_CHAT=$!
 
-    # look for the context size error in the server logs
-    if ! timeout 20 bash -c "
-        until grep -q 'exceed context window of' $TEST_CTX_SIZE_LAB_SERVE_LOG_FILE; do
-        echo 'waiting for context size error'
-        sleep 1
-    done
-"; then
-        echo "context size error not found in server logs"
-        cat $TEST_CTX_SIZE_LAB_SERVE_LOG_FILE
-        exit 1
-    fi
-
+    # we catch these failure BEOFRE they hit the server. as of llama_cpp_python 0.3.z, these types of failures are fatal and will crash the server
     # look for the context size error in the chat logs
     if ! timeout 20 bash -c "
         until grep -q 'Message too large for context size.' $TEST_CTX_SIZE_LAB_CHAT_LOG_FILE; do
@@ -560,7 +549,7 @@ test_server_chat_template() {
 }
 
 test_ilab_chat_server_logs(){
-    sed -i.bak '/max_ctx_size: 4096/s/4096/1/' "${ILAB_CONFIG_FILE}"
+    sed -i.bak '/max_ctx_size: 4096/s/4096/10/' "${ILAB_CONFIG_FILE}"
 
     chat_shot+=("--serving-log-file" "chat.log")
     "${chat_shot[@]}" "Hello"
@@ -570,7 +559,7 @@ test_ilab_chat_server_logs(){
             echo "waiting for chat log file to be created"
             sleep 1
         done
-        if ! grep "exceed context window of" chat.log; then
+        if ! grep "Message too large for context size" chat.log; then
             echo "chat log file does not contain serving message"
             exit 1
         else
diff --git a/tests/test_config.py b/tests/test_config.py
index 0f77e889cb..3cf95675e1 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -341,20 +341,19 @@ def test_logging(log_level, debug_level, root, instructlab, openai_httpx):
 @patch.multiple(
     config.DEFAULTS,
     _cache_home="/cache/instructlab",
-    _config_dir="/config/instructlab",
     _data_dir="/data/instructlab",
 )
 def test_compare_default_config_testdata(
-    testdata_path: pathlib.Path, tmp_path: pathlib.Path, regenerate_testdata: bool
+    testdata_path: pathlib.Path, regenerate_testdata: bool
 ):
     assert config.DEFAULTS.CHECKPOINTS_DIR == "/data/instructlab/checkpoints"
     saved_file = testdata_path / "default_config.yaml"
-    current_file = tmp_path / "current_config.yaml"
+    current_file = os.path.join(config.DEFAULTS._config_dir, "current_config.yaml")
 
     current_cfg = config.get_default_config()
     # roundtrip to verify serialization and de-serialization
     config.write_config(current_cfg, str(current_file))
-    with current_file.open(encoding="utf-8") as yamlfile:
+    with open(file=current_file, encoding="utf-8") as yamlfile:
         current_content = yaml.safe_load(yamlfile)
 
     if regenerate_testdata:
diff --git a/tests/testdata/default_config.yaml b/tests/testdata/default_config.yaml
index 0463050c44..61d7642c4c 100644
--- a/tests/testdata/default_config.yaml
+++ b/tests/testdata/default_config.yaml
@@ -170,8 +170,17 @@ generate:
     # Default: /cache/instructlab/models/granite-7b-lab-Q4_K_M.gguf
     model_path: /cache/instructlab/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf
     # Server configuration including host and port.
-    # Default: host='127.0.0.1' port=8000
+    # Default: host='127.0.0.1' port=8000 backend_type='' current_max_ctx_size=4096
     server:
+      # Backend Instance Type
+      # Default: ''
+      # Examples:
+      #   - llama-cpp
+      #   - vllm
+      backend_type: ''
+      # Maximum number of tokens that can be processed by the currently served model.
+      # Default: 4096
+      current_max_ctx_size: 4096
       # Host to serve on.
       # Default: 127.0.0.1
       host: 127.0.0.1
@@ -251,8 +260,17 @@ serve:
   # Default: /cache/instructlab/models/granite-7b-lab-Q4_K_M.gguf
   model_path: /cache/instructlab/models/granite-7b-lab-Q4_K_M.gguf
   # Server configuration including host and port.
-  # Default: host='127.0.0.1' port=8000
+  # Default: host='127.0.0.1' port=8000 backend_type='' current_max_ctx_size=4096
   server:
+    # Backend Instance Type
+    # Default: ''
+    # Examples:
+    #   - llama-cpp
+    #   - vllm
+    backend_type: ''
+    # Maximum number of tokens that can be processed by the currently served model.
+    # Default: 4096
+    current_max_ctx_size: 4096
     # Host to serve on.
     # Default: 127.0.0.1
     host: 127.0.0.1