text-generation-webui?
Hello and thanks for sharing your work
Do you know if this model can be used with textgeneration-webui? I cannot load it:
...
llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 323, got 291
llama_load_model_from_file: failed to load model
17:55:28-503397 ERROR Failed to load the model.
Traceback (most recent call last):
File "/opt/text-generation-webui/modules/ui_model_menu.py", line 245, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/text-generation-webui/modules/models.py", line 87, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/text-generation-webui/modules/models.py", line 250, in llamacpp_loader
model, tokenizer = LlamaCppModel.from_pretrained(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/text-generation-webui/modules/llamacpp_model.py", line 102, in from_pretrained
result.model = Llama(**params)
^^^^^^^^^^^^^^^
File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp_cuda/llama.py", line 311, in init
self._model = _LlamaModel(
^^^^^^^^^^^^
File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp_cuda/_internals.py", line 55, in init
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: models/medicine-chat.Q5_K_M.gguf
Exception ignored in: <function LlamaCppModel.__del__ at 0x7415bc831e40>
Traceback (most recent call last):
File "/opt/text-generation-webui/modules/llamacpp_model.py", line 58, in del
del self.model
^^^^^^^^^^
AttributeError: 'LlamaCppModel' object has no attribute 'model'
Any idea?
Thanks for reporting this
@blankreg
it's a bug in Llama.cpp in fact, I reported it here and hopefully we get it fixed: https://github.com/ggerganov/llama.cpp/issues/6490
So I should have reported it to github. Thanks for taking the time to open an issue with llama.cpp