Repository: ollama/ollama (Go)

Issue:
v0.3.7-rc5 no longer uses multiple GPUs for a single model
### What is the issue?

Moving from 0.3.6 to 0.3.7-rc5, Ollama no longer uses both GPUs for a single model when the model will not fit on one card.  If I load two models, though, it will use the second card to load the second model.  Output of "ollama ps" and "nvidia-smi" below.

```
ollama ps:
NAME            ID              SIZE    PROCESSOR       UNTIL
mistral-large:latest    0ca7dfa0bf06    71 GB   29%/71% CPU/GPU 4 minutes from now

nvidia-smi:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A6000               On  | 00000000:01:00.0 Off |                  Off |
| 37%   67C    P2             176W / 300W |  47681MiB / 49140MiB |     37%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000               On  | 00000000:E1:00.0 Off |                  Off |
| 30%   49C    P8              22W / 300W |      4MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    262888      C   ...unners/cuda_v11/ollama_llama_server    47674MiB |
+---------------------------------------------------------------------------------------+
```

And here's with two models loaded (but inactive):

```
NAME                    ID              SIZE    PROCESSOR       UNTIL
gemma2:27b              53261bc9c192    20 GB   100% GPU        4 minutes from now
mistral-large:latest    0ca7dfa0bf06    71 GB   29%/71% CPU/GPU 2 minutes from now

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A6000               On  | 00000000:01:00.0 Off |                  Off |
| 30%   51C    P8              25W / 300W |  47681MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000               On  | 00000000:E1:00.0 Off |                  Off |
| 37%   57C    P8              25W / 300W |  19053MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    262888      C   ...unners/cuda_v11/ollama_llama_server    47674MiB |
|    1   N/A  N/A    278516      C   ...unners/cuda_v11/ollama_llama_server    19046MiB |
+---------------------------------------------------------------------------------------+
```


### OS

Linux

### GPU

Nvidia

### CPU

AMD

### Ollama version

0.3.7-rc5


Discussion:
[Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may help in debugging.
```
[GIN] 2024/08/23 - 17:02:19 | 200 |       23.69µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/08/23 - 17:02:19 | 200 |    3.728994ms |       127.0.0.1 | POST     "/api/show"
time=2024-08-23T17:02:20.155Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=89 layers.offload=62 l



















                                                                                                                                          layers.split="" memory.available="[47.3 GiB]" memory.required.full="66.5 GiB" memory.required.partial="47.2 GiB" memory.required.kv="704.0 MiB" memory.required.allocations="[47.2 GiB]" memory.weights.total="64.5 GiB" memory.weights.repeating="64.2 GiB" memory.weights.nonrepeating="315.0 MiB" memory.graph.full="484.0 MiB" memory.graph.partial="545.0 MiB"
time=2024-08-23T17:02:20.157Z level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama3372309948/runners/cuda_v11/ollama_llama_server --model /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 62 --parallel 1 --port 37647"
time=2024-08-23T17:02:20.157Z level=INFO source=sched.go:445 msg="loaded runners" count=1
time=2024-08-23T17:02:20.157Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
time=2024-08-23T17:02:20.157Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="1e6f655" tid="123602874314752" timestamp=1724432540
INFO [main] system info | n_threads=24 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="123602874314752" timestamp=1724432540 total_threads=48
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="47" port="37647" tid="123602874314752" timestamp=1724432540
llama_model_loader: loaded meta data with 34 key-value pairs and 795 tensors from /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Mistral Large Instruct 2407
llama_model_loader: - kv   3:                            general.version str              = 2407
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = Mistral
llama_model_loader: - kv   6:                         general.size_label str              = Large
llama_model_loader: - kv   7:                            general.license str              = other
llama_model_loader: - kv   8:                       general.license.name str              = mrl
llama_model_loader: - kv   9:                       general.license.link str              = https://mistral.ai/licenses/MRL-0.1.md
llama_model_loader: - kv  10:                          general.languages arr[str,10]      = ["en", "fr", "de", "es", "it", "pt", ...
llama_model_loader: - kv  11:                          llama.block_count u32              = 88
llama_model_loader: - kv  12:                       llama.context_length u32              = 32768
llama_model_loader: - kv  13:                     llama.embedding_length u32              = 12288
llama_model_loader: - kv  14:                  llama.feed_forward_length u32              = 28672
llama_model_loader: - kv  15:                 llama.attention.head_count u32              = 96
llama_model_loader: - kv  16:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  17:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  19:                          general.file_type u32              = 2
llama_model_loader: - kv  20:                           llama.vocab_size u32              = 32768
llama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  22:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,32768]   = ["<unk>", "
", "
", "
", "[...
llama_model_loader: - kv  26:                      tokenizer.ggml.scores arr[f32,32768]   = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,32768]   = [3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, ...
llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  30:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  32:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  177 tensors
llama_model_loader: - type q4_0:  617 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 771
llm_load_vocab: token to piece cache size = 0.1732 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32768
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 12288
llm_load_print_meta: n_layer          = 88
llm_load_print_meta: n_head           = 96
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 12
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 28672
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 122.61 B
llm_load_print_meta: model size       = 64.34 GiB (4.51 BPW)
llm_load_print_meta: general.name     = Mistral Large Instruct 2407
llm_load_print_meta: BOS token        = 1 '
'
llm_load_print_meta: EOS token        = 2 '
'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 781 '<0x0A>'
llm_load_print_meta: max token length = 48
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.74 MiB
time=2024-08-23T17:02:20.608Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
time=2024-08-23T17:02:23.565Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
llm_load_tensors: offloading 62 repeating layers to GPU
llm_load_tensors: offloaded 62/89 layers to GPU
llm_load_tensors:        CPU buffer size = 65879.30 MiB
llm_load_tensors:      CUDA0 buffer size = 46040.81 MiB
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =   208.00 MiB
llama_kv_cache_init:      CUDA0 KV buffer size =   496.00 MiB
llama_new_context_with_model: KV self size  =  704.00 MiB, K (f16):  352.00 MiB, V (f16):  352.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.17 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   513.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    28.01 MiB
llama_new_context_with_model: graph nodes  = 2822
llama_new_context_with_model: graph splits = 290
INFO [main] model loaded | tid="123602874314752" timestamp=1724432546
time=2024-08-23T17:02:26.576Z level=INFO source=server.go:630 msg="llama runner started in 6.42 seconds"
[GIN] 2024/08/23 - 17:02:26 | 200 |  6.603581689s |       127.0.0.1 | POST     "/api/chat"

```
If you could add `OLLAMA_DEBUG=1` to the server environment, restart and load the model and then post the full log, that will be helpful.
```
[GIN] 2024/08/23 - 17:58:33 | 200 |       45.17µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/08/23 - 17:58:33 | 200 |    3.617657ms |       127.0.0.1 | POST     "/api/show"
time=2024-08-23T17:58:33.381Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="251.2 GiB" before.free="247.0 G



















                                                                                                                                          GiB" before.free_swap="64.0 GiB" now.total="251.2 GiB" now.free="246.8 GiB" now.free_swap="64.0 GiB"
CUDA driver version: 12.2
time=2024-08-23T17:58:33.457Z level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55 name="NVIDIA RTX A6000" overhead="0 B" before.total="47.5 GiB" before.free="47.3 GiB" now.total="47.5 GiB" now.free="47.3 GiB" now.used="266.6 MiB"
time=2024-08-23T17:58:33.528Z level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877 name="NVIDIA RTX A6000" overhead="0 B" before.total="47.5 GiB" before.free="47.3 GiB" now.total="47.5 GiB" now.free="47.3 GiB" now.used="266.6 MiB"
releasing cuda driver library
time=2024-08-23T17:58:33.528Z level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x8161c0 gpu_count=2
time=2024-08-23T17:58:33.531Z level=DEBUG source=sched.go:219 msg="loading first model" model=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822
time=2024-08-23T17:58:33.531Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.532Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.532Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.533Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.534Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.535Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.535Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.536Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.536Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.537Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.537Z level=DEBUG source=server.go:101 msg="system memory" total="251.2 GiB" free="246.8 GiB" free_swap="64.0 GiB"
time=2024-08-23T17:58:33.537Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.538Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=89 layers.offload=62 layers.split="" memory.available="[47.3 GiB]" memory.required.full="66.5 GiB" memory.required.partial="47.2 GiB" memory.required.kv="704.0 MiB" memory.required.allocations="[47.2 GiB]" memory.weights.total="64.5 GiB" memory.weights.repeating="64.2 GiB" memory.weights.nonrepeating="315.0 MiB" memory.graph.full="484.0 MiB" memory.graph.partial="545.0 MiB"
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx2/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v11/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v12/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/rocm_v60102/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx2/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v11/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v12/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/rocm_v60102/ollama_llama_server
time=2024-08-23T17:58:33.541Z level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama1191698724/runners/cuda_v11/ollama_llama_server --model /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 62 --verbose --parallel 1 --port 34631"
time=2024-08-23T17:58:33.541Z level=DEBUG source=server.go:408 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin LD_LIBRARY_PATH=/home/dfm/ollama/lib/ollama:/tmp/ollama1191698724/runners/cuda_v11 CUDA_VISIBLE_DEVICES=GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55]"
time=2024-08-23T17:58:33.541Z level=INFO source=sched.go:445 msg="loaded runners" count=1
time=2024-08-23T17:58:33.541Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
time=2024-08-23T17:58:33.542Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="1e6f655" tid="129151660158976" timestamp=1724435913
INFO [main] system info | n_threads=24 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="129151660158976" timestamp=1724435913 total_threads=48
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="47" port="34631" tid="129151660158976" timestamp=1724435913
llama_model_loader: loaded meta data with 34 key-value pairs and 795 tensors from /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Mistral Large Instruct 2407
llama_model_loader: - kv   3:                            general.version str              = 2407
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = Mistral
llama_model_loader: - kv   6:                         general.size_label str              = Large
llama_model_loader: - kv   7:                            general.license str              = other
llama_model_loader: - kv   8:                       general.license.name str              = mrl
llama_model_loader: - kv   9:                       general.license.link str              = https://mistral.ai/licenses/MRL-0.1.md
llama_model_loader: - kv  10:                          general.languages arr[str,10]      = ["en", "fr", "de", "es", "it", "pt", ...
llama_model_loader: - kv  11:                          llama.block_count u32              = 88
llama_model_loader: - kv  12:                       llama.context_length u32              = 32768
llama_model_loader: - kv  13:                     llama.embedding_length u32              = 12288
llama_model_loader: - kv  14:                  llama.feed_forward_length u32              = 28672
llama_model_loader: - kv  15:                 llama.attention.head_count u32              = 96
llama_model_loader: - kv  16:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  17:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  19:                          general.file_type u32              = 2
llama_model_loader: - kv  20:                           llama.vocab_size u32              = 32768
llama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  22:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,32768]   = ["<unk>", "
", "
", "
", "[...
llama_model_loader: - kv  26:                      tokenizer.ggml.scores arr[f32,32768]   = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,32768]   = [3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, ...
llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  30:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  32:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  177 tensors
llama_model_loader: - type q4_0:  617 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 771
llm_load_vocab: token to piece cache size = 0.1732 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32768
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 12288
llm_load_print_meta: n_layer          = 88
llm_load_print_meta: n_head           = 96
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 12
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 28672
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 122.61 B
llm_load_print_meta: model size       = 64.34 GiB (4.51 BPW)
llm_load_print_meta: general.name     = Mistral Large Instruct 2407
llm_load_print_meta: BOS token        = 1 '
'
llm_load_print_meta: EOS token        = 2 '
'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 781 '<0x0A>'
llm_load_print_meta: max token length = 48
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.74 MiB
time=2024-08-23T17:58:33.993Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
time=2024-08-23T17:58:37.143Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
llm_load_tensors: offloading 62 repeating layers to GPU
llm_load_tensors: offloaded 62/89 layers to GPU
llm_load_tensors:        CPU buffer size = 65879.30 MiB
llm_load_tensors:      CUDA0 buffer size = 46040.81 MiB
time=2024-08-23T17:58:37.394Z level=DEBUG source=server.go:636 msg="model load progress 0.33"
time=2024-08-23T17:58:37.645Z level=DEBUG source=server.go:636 msg="model load progress 0.42"
time=2024-08-23T17:58:37.896Z level=DEBUG source=server.go:636 msg="model load progress 0.51"
time=2024-08-23T17:58:38.147Z level=DEBUG source=server.go:636 msg="model load progress 0.60"
time=2024-08-23T17:58:38.398Z level=DEBUG source=server.go:636 msg="model load progress 0.70"
time=2024-08-23T17:58:38.649Z level=DEBUG source=server.go:636 msg="model load progress 0.79"
time=2024-08-23T17:58:38.900Z level=DEBUG source=server.go:636 msg="model load progress 0.88"
time=2024-08-23T17:58:39.151Z level=DEBUG source=server.go:636 msg="model load progress 0.97"
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =   208.00 MiB
llama_kv_cache_init:      CUDA0 KV buffer size =   496.00 MiB
llama_new_context_with_model: KV self size  =  704.00 MiB, K (f16):  352.00 MiB, V (f16):  352.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.17 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   513.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    28.01 MiB
llama_new_context_with_model: graph nodes  = 2822
llama_new_context_with_model: graph splits = 290
time=2024-08-23T17:58:39.401Z level=DEBUG source=server.go:636 msg="model load progress 1.00"
time=2024-08-23T17:58:39.652Z level=DEBUG source=server.go:639 msg="model load completed, waiting for server to become available" status="llm server loading model"
DEBUG [initialize] initializing slots | n_slots=1 tid="129151660158976" timestamp=1724435920
DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=0 tid="129151660158976" timestamp=1724435920
INFO [main] model loaded | tid="129151660158976" timestamp=1724435920
DEBUG [update_slots] all slots are idle and system prompt is empty, clear the KV cache | tid="129151660158976" timestamp=1724435920
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=0 tid="129151660158976" timestamp=1724435920
time=2024-08-23T17:58:40.154Z level=INFO source=server.go:630 msg="llama runner started in 6.61 seconds"
time=2024-08-23T17:58:40.154Z level=DEBUG source=sched.go:458 msg="finished setting up runner" model=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822
[GIN] 2024/08/23 - 17:58:40 | 200 |  6.777621249s |       127.0.0.1 | POST     "/api/chat"
time=2024-08-23T17:58:40.155Z level=DEBUG source=sched.go:462 msg="context for request finished"
time=2024-08-23T17:58:40.155Z level=DEBUG source=sched.go:334 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 duration=5m0s
time=2024-08-23T17:58:40.155Z level=DEBUG source=sched.go:352 msg="after processing request finished event" modelPath=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 refCount=0

```
Please post the start of the log.
Third time's the charm I guess...  lol  It does see both cards, and uses both when I load two models.  That's why I limited it to just the load part of the log, but here's the log from server launch to just before I ran the model that resulted in the log above.

```
2024/08/23 18:06:38 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/dfm/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-08-23T18:06:38.408Z level=INFO source=images.go:752 msg="total blobs: 35"
time=2024-08-23T18:06:38.408Z level=INFO source=images.go:759 msg="total unused blobs removed: 0"
time=2024-08-23T18:06:38.409Z level=INFO source=routes.go:1172 msg="Listening on 127.0.0.1:11434 (version 0.3.7-rc5)"
time=2024-08-23T18:06:38.409Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3793318785/runners
time=2024-08-23T18:06:38.409Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/ollama_llama_server.gz
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cpu/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cpu_avx/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cpu_avx2/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cuda_v11/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cuda_v12/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/rocm_v60102/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102]"
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-08-23T18:06:43.205Z level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2024-08-23T18:06:43.205Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
time=2024-08-23T18:06:43.205Z level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA"
time=2024-08-23T18:06:43.205Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so*
time=2024-08-23T18:06:43.205Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/home/dfm/ollama/lib/ollama/libcuda.so* /home/dfm/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2024-08-23T18:06:43.206Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.535.183.01]
CUDA driver version: 12.2
time=2024-08-23T18:06:46.702Z level=DEBUG source=gpu.go:119 msg="detected GPUs" count=2 library=/usr/lib/x86_64-linux-gnu/libcuda.so.535.183.01
[GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55] CUDA totalMem 48676 mb
[GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55] CUDA freeMem 48410 mb
[GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55] Compute Capability 8.6
[GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877] CUDA totalMem 48676 mb
[GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877] CUDA freeMem 48410 mb
[GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877] Compute Capability 8.6
time=2024-08-23T18:06:47.026Z level=DEBUG source=amd_linux.go:371 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2024-08-23T18:06:47.026Z level=INFO source=types.go:107 msg="inference compute" id=GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55 library=cuda variant=v11 compute=8.6 driver=12.2 name="NVIDIA RTX A6000" total="47.5 GiB" available="47.3 GiB"
time=2024-08-23T18:06:47.026Z level=INFO source=types.go:107 msg="inference compute" id=GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877 library=cuda variant=v11 compute=8.6 driver=12.2 name="NVIDIA RTX A6000" total="47.5 GiB" available="47.3 GiB"

```
I've repro'd.  We'll get a fix for this before shipping 0.3.7.  Thanks!



Patch:
diff --git a/gpu/types.go b/gpu/types.go
index 4cbbeb841b5..a30e5fb3624 100644
--- a/gpu/types.go
+++ b/gpu/types.go
@@ -94,7 +94,7 @@ func (l GpuInfoList) ByLibrary() []GpuInfoList {
 			}
 		}
 		if !found {
-			libs = append(libs, info.Library)
+			libs = append(libs, requested)
 			resp = append(resp, []GpuInfo{info})
 		}
 	}


Test patch:
diff --git a/gpu/gpu_test.go b/gpu/gpu_test.go
index 46d3201e1c8..13a3f544210 100644
--- a/gpu/gpu_test.go
+++ b/gpu/gpu_test.go
@@ -32,4 +32,29 @@ func TestCPUMemInfo(t *testing.T) {
 	}
 }
 
+func TestByLibrary(t *testing.T) {
+	type testCase struct {
+		input  []GpuInfo
+		expect int
+	}
+
+	testCases := map[string]*testCase{
+		"empty":                    {input: []GpuInfo{}, expect: 0},
+		"cpu":                      {input: []GpuInfo{{Library: "cpu"}}, expect: 1},
+		"cpu + GPU":                {input: []GpuInfo{{Library: "cpu"}, {Library: "cuda"}}, expect: 2},
+		"cpu + 2 GPU no variant":   {input: []GpuInfo{{Library: "cpu"}, {Library: "cuda"}, {Library: "cuda"}}, expect: 2},
+		"cpu + 2 GPU same variant": {input: []GpuInfo{{Library: "cpu"}, {Library: "cuda", Variant: "v11"}, {Library: "cuda", Variant: "v11"}}, expect: 2},
+		"cpu + 2 GPU diff variant": {input: []GpuInfo{{Library: "cpu"}, {Library: "cuda", Variant: "v11"}, {Library: "cuda", Variant: "v12"}}, expect: 3},
+	}
+
+	for k, v := range testCases {
+		t.Run(k, func(t *testing.T) {
+			resp := (GpuInfoList)(v.input).ByLibrary()
+			if len(resp) != v.expect {
+				t.Fatalf("expected length %d, got %d => %+v", v.expect, len(resp), resp)
+			}
+		})
+	}
+}
+
 // TODO - add some logic to figure out card type through other means and actually verify we got back what we expected