Repository: abetlen/llama-cpp-python

Issue #459: CMakeDetermineCUDACompiler.cmake:227 (message): Couldn't find CUDA library root
Created: 2023-07-09T13:20:57Z
Prerequisites
Please answer the following questions for yourself before submitting an issue.

 I am running the latest code. Development is very rapid so there are no tagged versions as of now.
 I carefully followed the README.md.
 I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
 I reviewed the Discussions, and have a new bug or useful enhancement to share.

Issue at hand
Running Ubunt 22.04 with CUDA 12 installed from NVIDIA at /usr/local/cuda-12.2, the installation is not able to find the CUDA library root. What could be going wrong?
$ CMAKE_ARGS="-DLLAMA_CUBLAS=on -DCUDA_PATH=/usr/local/cuda-12.2 -DCUDAToolkit_ROOT=/usr/local/cuda-12.2" FORCE_CMAKE=1 CUDA_PATH=/usr/local/cuda-12.2 CUDAToolkit_ROOT=/usr/local/cuda-12.2 pip install llama-cpp-python --no-cache-dir
Collecting llama-cpp-python
  Downloading llama_cpp_python-0.1.68.tar.gz (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 10.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.5.0 in ./installer_files/env/lib/python3.10/site-packages (from llama-cpp-python) (4.7.1)
Requirement already satisfied: numpy>=1.20.0 in ./installer_files/env/lib/python3.10/site-packages (from llama-cpp-python) (1.24.4)
Requirement already satisfied: diskcache>=5.6.1 in ./installer_files/env/lib/python3.10/site-packages (from llama-cpp-python) (5.6.1)
Building wheels for collected packages: llama-cpp-python
  Building wheel for llama-cpp-python (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for llama-cpp-python (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [91 lines of output]
      
      
      --------------------------------------------------------------------------------
      -- Trying 'Ninja' generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      Not searching for unused variables given on the command line.
      -- The C compiler identification is GNU 11.3.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- The CXX compiler identification is GNU 11.3.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Configuring done (0.2s)
      -- Generating done (0.0s)
      -- Build files have been written to: /tmp/pip-install-3gz5ypww/llama-cpp-python_983973d9dc5f427c90a4917b80c23118/_cmake_test_compile/build
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying 'Ninja' generator - success
      --------------------------------------------------------------------------------
      
      Configuring Project
        Working directory:
          /tmp/pip-install-3gz5ypww/llama-cpp-python_983973d9dc5f427c90a4917b80c23118/_skbuild/linux-x86_64-3.10/cmake-build
        Command:
          /tmp/pip-build-env-gq0ai9gf/overlay/lib/python3.10/site-packages/cmake/data/bin/cmake /tmp/pip-install-3gz5ypww/llama-cpp-python_983973d9dc5f427c90a4917b80c23118 -G Ninja -DCMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-gq0ai9gf/overlay/lib/python3.10/site-packages/ninja/data/bin/ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-3gz5ypww/llama-cpp-python_983973d9dc5f427c90a4917b80c23118/_skbuild/linux-x86_64-3.10/cmake-install -DPYTHON_VERSION_STRING:STRING=3.10.12 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-gq0ai9gf/overlay/lib/python3.10/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/home/vadi/Programs/oobabooga_linux/installer_files/env/bin/python -DPYTHON_INCLUDE_DIR:PATH=/home/vadi/Programs/oobabooga_linux/installer_files/env/include/python3.10 -DPYTHON_LIBRARY:PATH=/home/vadi/Programs/oobabooga_linux/installer_files/env/lib/libpython3.10.so -DPython_EXECUTABLE:PATH=/home/vadi/Programs/oobabooga_linux/installer_files/env/bin/python -DPython_ROOT_DIR:PATH=/home/vadi/Programs/oobabooga_linux/installer_files/env -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/home/vadi/Programs/oobabooga_linux/installer_files/env/include/python3.10 -DPython3_EXECUTABLE:PATH=/home/vadi/Programs/oobabooga_linux/installer_files/env/bin/python -DPython3_ROOT_DIR:PATH=/home/vadi/Programs/oobabooga_linux/installer_files/env -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/home/vadi/Programs/oobabooga_linux/installer_files/env/include/python3.10 -DCMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-gq0ai9gf/overlay/lib/python3.10/site-packages/ninja/data/bin/ninja -DLLAMA_CUBLAS=on -DCUDA_PATH=/usr/local/cuda-12.2 -DCUDAToolkit_ROOT=/usr/local/cuda-12.2 -DCMAKE_BUILD_TYPE:STRING=Release -DLLAMA_CUBLAS=on -DCUDA_PATH=/usr/local/cuda-12.2 -DCUDAToolkit_ROOT=/usr/local/cuda-12.2
      
      Not searching for unused variables given on the command line.
      -- The C compiler identification is GNU 11.3.0
      -- The CXX compiler identification is GNU 11.3.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found Git: /home/vadi/Programs/oobabooga_linux/installer_files/env/bin/git (found version "2.40.1")
      fatal: not a git repository (or any of the parent directories): .git
      fatal: not a git repository (or any of the parent directories): .git
      CMake Warning at vendor/llama.cpp/CMakeLists.txt:114 (message):
        Git repository not found; to enable automatic generation of build info,
        make sure Git is installed and the project is a Git repository.
      
      
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
      -- Found Threads: TRUE
      -- Found CUDAToolkit: /usr/local/cuda-12.2/include (found version "12.2.91")
      -- cuBLAS found
      CMake Error at /tmp/pip-build-env-gq0ai9gf/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeDetermineCUDACompiler.cmake:227 (message):
        Couldn't find CUDA library root.
      Call Stack (most recent call first):

Duplicate issue #409: Could not find nvcc, please set CUDAToolkit_ROOT
Created: 2023-06-20T22:34:34Z
Prerequisites
Please answer the following questions for yourself before submitting an issue.

[ X] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
[ X] I carefully followed the README.md.
[ X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
[ X] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior
--n-gpu-layers 36 is supposed to fill my VRAM and use my GPU, it's also supposed to print in the console llama_model_load_internal: [cublas] offloading 36 layers to GPU
and I suppose it should be printing BLAS = 1
Current Behavior
I don't have anything about offloading in the console, my GPU is sleeping, and my VRAM is empty. And it prints BLAS = 0
Environment and Context
Windows 10
CPU with 20 threads
64 Go RAM
RTX 3080 Ti Laptop GPU
16 Go VRAM
Python 3.10.9
fastapi 0.97.0
numpy 1.25.0
starlette 0.27.0
uvicorn 0.22.0
Can't verify "make" and "g++" versions because windows does not find these commands
Steps to Reproduce

have my gpu (maybe it's an issue with my gpu)
install oobabooga's text-generation-webui https://github.com/oobabooga/text-generation-webui

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui

conda create -n textgen python=3.10.9
conda activate textgen
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117
pip install -r requirements.txt

pip uninstall -y llama-cpp-python
set CMAKE_ARGS="-DLLAMA_CUBLAS=on"
set FORCE_CMAKE=1
pip install llama-cpp-python --no-cache-dir


download a ggml model (I used WizardLM-30B-Uncensored.ggmlv3.q4_0.bin) and put it in the "models" folder
start the webui

python server.py --model WizardLM-30B-Uncensored.ggmlv3.q4_0.bin  --n-gpu-layers 36 --auto-devices


open the gradio webui
generate something in the chat-box to load the model

Failure Logs
bin C:\Users\Armaguedin\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.dll
C:\Users\Armaguedin\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
function 'cadam32bit_grad_fp32' not found
2023-06-20 23:40:24 INFO:Loading WizardLM-30B-Uncensored.ggmlv3.q4_0.bin...
2023-06-20 23:40:24 INFO:llama.cpp weights detected: models\WizardLM-30B-Uncensored.ggmlv3.q4_0.bin

2023-06-20 23:40:24 INFO:Cache capacity is 0 bytes
llama.cpp: loading model from models\WizardLM-30B-Uncensored.ggmlv3.q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32001
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 6656
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 52
llama_model_load_internal: n_layer    = 60
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 17920
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size =    0.13 MB
llama_model_load_internal: mem required  = 19756.67 MB (+ 3124.00 MB per state)
....................................................................................................
llama_init_from_file: kv self size  = 3120.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
2023-06-20 23:40:25 INFO:Loaded the model in 0.97 seconds.

2023-06-20 23:40:25 INFO:Loading the extension "gallery"...
Running on local URL:  http://0.0.0.0:7861

To create a public link, set `share=True` in `launch()`.

llama_print_timings:        load time = 60209.82 ms
llama_print_timings:      sample time =     1.47 ms /    10 runs   (    0.15 ms per token,  6807.35 tokens per second)
llama_print_timings: prompt eval time = 60209.69 ms /     8 tokens ( 7526.21 ms per token,     0.13 tokens per second)
llama_print_timings:        eval time =  8880.62 ms /     9 runs   (  986.74 ms per token,     1.01 tokens per second)
llama_print_timings:       total time = 69109.63 ms
Output generated in 69.34 seconds (0.13 tokens/s, 9 tokens, context 8, seed 1976462288)

Closing comment (2023-07-10T06:57:53Z): Likely a duplicate of #459. You need to fix your CUDA install.