Repository: mudler/LocalAI

Issue #2258: Mode tts, gpt4 or vision frozes from time to time without timeout
Created: 2024-05-07T13:46:08Z
LocalAI version:
2.14.0
Environment, CPU architecture, OS, and Version:
Linux Ubuntu SMP PREEMPT_DYNAMIC  x86_64 x86_64 x86_64 GNU/Linux
90GB RAM 22 vcores
nvidia L4 24GB
Describe the bug
Requests frozen from time to time
My logs are continuously producing:
2024-05-06T14:09:55.116298748Z 2:09PM DBG GRPC(Llama3-8B-OpenHermes-DPO.Q4_K_M.gguf-127.0.0.1:44249): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
api_1  | 2024-05-06T14:09:55.116344987Z 2:09PM DBG GRPC(Llama3-8B-OpenHermes-DPO.Q4_K_M.gguf-127.0.0.1:44249): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
api_1  | 2024-05-06T14:09:55.116358667Z 2:09PM DBG GRPC(Llama3-8B-OpenHermes-DPO.Q4_K_M.gguf-127.0.0.1:44249): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64
api_1  | 2024-05-06T14:09:55.116369597Z 2:09PM DBG GRPC(Llama3-8B-OpenHermes-DPO.Q4_K_M.gguf-127.0.0.1:44249): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32
api_1  | 2024-05-06T14:09:55.116380867Z 2:09PM DBG GRPC(Llama3-8B-OpenHermes-DPO.Q4_K_M.gguf-127.0.0.1:44249): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 16
api_1  | 2024-05-06T14:09:55.116394727Z 2:09PM DBG GRPC(Llama3-8B-OpenHermes-DPO.Q4_K_M.gguf-127.0.0.1:44249): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 8
api_1  | 2024-05-06T14:09:55.116405866Z 2:09PM DBG GRPC(Llama3-8B-OpenHermes-DPO.Q4_K_M.gguf-127.0.0.1:44249): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 4
api_1  | 2024-05-06T14:09:55.116433847Z 2:09PM DBG GRPC(Llama3-8B-OpenHermes-DPO.Q4_K_M.gguf-127.0.0.1:44249): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 2
api_1  | 2024-05-06T14:09:55.116445007Z 2:09PM DBG GRPC(Llama3-8B-OpenHermes-DPO.Q4_K_M.gguf-127.0.0.1:44249): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 1

Duplicate issue #2282: ERROR: stderr update_slots : failed to find free space in the KV cache
Created: 2024-05-10T06:20:32Z
LocalAI version:
v2.14.0
Environment, CPU architecture, OS, and Version:

Apple MacBook M2 Max
MacOS Ventura 13.6.3 (22G436)

Describe the bug
Errors, when trying to process request.
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32

To Reproduce

URL: /v1/chat/completions
Model: orca-2-13b-q4.gguf (https://huggingface.co/TheBloke/Orca-2-13B-GGUF/resolve/main/orca-2-13b.Q4_0.gguf)
orca-2-13b-q4.yaml

#######################
# orca-2-13b-q4 #
#######################

# @see https://huggingface.co/TheBloke/Orca-2-13B-GGUF/resolve/main/orca-2-13b.Q4_0.gguf
context_size: 2000
f16: true
# gpu_layers: 1
gpu_layers: 50
name: orca-2-13b-q4
parameters:
  model: orca-2-13b-q4.gguf
  temperature: 0.9
  top_k: 40
  top_p: 0.65
embeddings: true


orca-2-13b-q4.gguf.tmpl


system
Answer the question. Be Concise.


user
{{.Input}}


assistant


Request:

{
  "model": "orca-2-13b-q4",
  "language": "",
  "n": 1,
  "top_p": 1,
  "top_k": null,
  "temperature": 0,
  "max_tokens": null,
  "echo": false,
  "batch": 0,
  "ignore_eos": false,
  "repeat_penalty": 0,
  "n_keep": 0,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "tfz": null,
  "typical_p": null,
  "seed": null,
  "negative_prompt": "",
  "rope_freq_base": 0,
  "rope_freq_scale": 0,
  "negative_prompt_scale": 0,
  "use_fast_tokenizer": false,
  "clip_skip": 0,
  "tokenizer": "",
  "file": "",
  "response_format": {},
  "size": "",
  "prompt": null,
  "instruction": "",
  "input": null,
  "stop": null,
  "messages": [
    {
      "role": "system",
      "content": "You need to correct the errors in the text after the OCR. Return only corrected text."
    },
    {
      "role": "user",
      "content": "XXX XXXXXX XXXXX XXXXXXX mollell vubgl Wguw\n\nCOMMERCIAL LICENSE ﺔﻳﺭﺎﺠﺗ ﺔﺼﺧﺭ\n\nCompany : XXXXXXXXXXX\n\n: ﺔﻛﺮﺸﻟﺍ ﻢﺳﺍ\n\nﺪﺘﻴﻤﻴﻟ ﺏﻭﺮﺟ ﺵﺮﻴﺴﻳﺭ ﺲﺘﻴﻛﺭﺎﻣ ﻞﻴﻔﻴﻟ\n\n: ﻞﻴﺠﺴﺘﻟﺍ ﻢﻗﺭ\n\nRegistered Number :\n\n1111111111\n\nType of Legal Entity : Private Company Limited by Shares\n\n: xxxxxxxxx\n\nAddress : Dedicated Desk 1111, Floor 11, XX Xxxxx Xxxxx, Xxx Xxxx Xxxx Xxxx Xxxxx, Xxx Xxxxxx Xxxx, Xxx Xxxx, \n\n ,1119 \n\nAuthorised Signatory : Xxx Xxxx\n\n: \n\n\n\nBusiness Activities : Market research and public opinion polling ; Other information service activities n.e.c ; Management consultancy activities ; Advertising\n\n: ﺔ\n\nIssue Date :\n\n14 October 2010\n\n: ﺭﺍﺪﺻﻹﺍ ﺦﻳﺭﺎﺗ\n\nExpiry Date :\n\n13 October 2010\n\n: \n\nVerify Document Code COMPANIES-12121212\n\nApproved Electronic Document issued by Xxx Xxxxx Xxxxxx Xxxxxx Xxxxxx Xxxxxx. To verify, please visit www.xxxxxx.yyyyyy.com\n"
    }
  ],
  "functions": null,
  "function_call": null,
  "stream": false,
  "mode": 0,
  "step": 0,
  "grammar": "",
  "grammar_json_functions": null,
  "backend": "",
  "model_base_name": ""
}
Expected behavior
Some text response.
Logs
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to decode the batch, n_batch = 1, ret = 1
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 16
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 8
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 4
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 2
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 1
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to decode the batch, n_batch = 1, ret = 1
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
10:05AM DBG GRPC(orca-2-13b-q4.gguf-127.0.0.1:51965): stderr update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64

Additional context

Closing comment (2024-05-10T10:38:46Z): related to #2258