Kernel crash when using llama2 70b on langchain with llama.cpp
Labels: name: auto:bug, name: area: models

### System Info

Mac OS 14.0
M1 Max 64GB ram

VSCode 1.80.2
Jupyter Notebook

Python 3.11.4
Llama-cpp-python using `!CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install --upgrade llama-cpp-python`



### Who can help?

@hwchase17 @agol

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain.llms import LlamaCpp
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

llm = LlamaCpp(
    model_path="./llama2_70b_chat_uncensored.ggmlv3.q5_K_S.bin",
    n_gpu_layers=n_gpu_layers,
    n_gqa=8,
    n_batch=n_batch,
    n_ctx=2048,
    f16_kv=True,  
    callback_manager=callback_manager,
    verbose=True,
)
prompt = """
Question: A rap battle between Stephen Colbert and John Oliver
"""
llm(prompt)
```

Error log:
```
10:30:36.498 [error] Disposing session as kernel process died ExitCode: undefined, Reason: 0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
llama.cpp: loading model from ./llama2_70b_chat_uncensored.ggmlv3.q5_K_S.bin
llama_model_load_internal: warning: assuming 70B model based on GQA == 8
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 8192
llama_model_load_internal: n_mult     = 7168
llama_model_load_internal: n_head     = 64
llama_model_load_internal: n_head_kv  = 8
llama_model_load_internal: n_layer    = 80
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 8
llama_model_load_internal: rnorm_eps  = 1.0e-06
llama_model_load_internal: n_ff       = 28672
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 16 (mostly Q5_K - Small)
llama_model_load_internal: model size = 70B
llama_model_load_internal: ggml ctx size =    0.21 MB
llama_model_load_internal: mem required  = 46046.21 MB (+  640.00 MB per state)
llama_new_context_with_model: kv self size  =  640.00 MB
ggml_metal_init: allocating
ggml_metal_init: using MPS
ggml_metal_init: loading '~/anaconda3/envs/llama-cpp-venv/lib/python3.11/site-packages/llama_cpp/ggml-metal.metal'
ggml_metal_init: loaded kernel_add                            0x126d41720
ggml_metal_init: loaded kernel_add_row                        0x126d41130
ggml_metal_init: loaded kernel_mul                            0x126d4e410
ggml_metal_init: loaded kernel_mul_row                        0x126d4e710
ggml_metal_init: loaded kernel_scale                          0x126d55390
ggml_metal_init: loaded kernel_silu                           0x126d538c0
ggml_metal_init: loaded kernel_relu                           0x126d56130
ggml_metal_init: loaded kernel_gelu                           0x126d568e0
ggml_metal_init: loaded kernel_soft_max                       0x126d56d00
ggml_metal_init: loaded kernel_diag_mask_inf                  0x126d56f70
ggml_metal_init: loaded kernel_get_rows_f16                   0x126d57af0
ggml_metal_init: loaded kernel_get_rows_q4_0                  0x126d581f0
ggml_metal_init: loaded kernel_get_rows_q4_1                  0x126d58780
ggml_metal_init: loaded kernel_get_rows_q2_K                  0x111ce66e0
ggml_metal_init: loaded kernel_get_rows_q3_K                  0x126d589f0
ggml_metal_init: loaded kernel_get_rows_q4_K                  0x126d58c60
ggml_metal_init: loaded kernel_get_rows_q5_K                  0x126d59830
ggml_metal_init: loaded kernel_get_rows_q6_K                  0x126d59d80
ggml_metal_init: loaded kernel_rms_norm                       0x130328d00
ggml_metal_init: loaded kernel_norm                           0x1303284a0
ggml_metal_init: loaded kernel_mul_mat_f16_f32                0x13032a1c0
ggml_metal_init: loaded kernel_mul_mat_q4_0_f32               0x126d59ff0
ggml_metal_init: loaded kernel_mul_mat_q4_1_f32               0x126d5a260
ggml_metal_init: loaded kernel_mul_mat_q2_K_f32               0x126d5b0c0
ggml_metal_init: loaded kernel_mul_mat_q3_K_f32               0x126d5b640
ggml_metal_init: loaded kernel_mul_mat_q4_K_f32               0x126d5bc50
ggml_metal_init: loaded kernel_mul_mat_q5_K_f32               0x126d5c240
ggml_metal_init: loaded kernel_mul_mat_q6_K_f32               0x126d5c960
ggml_metal_init: loaded kernel_rope                           0x13032af10
ggml_metal_init: loaded kernel_alibi_f32                      0x13032b180
ggml_metal_init: loaded kernel_cpy_f32_f16                    0x13032b3f0
ggml_metal_init: loaded kernel_cpy_f32_f32                    0x13032b660
ggml_metal_init: loaded kernel_cpy_f16_f16                    0x13032d380
ggml_metal_init: recommendedMaxWorkingSetSize = 49152.00 MB
ggml_metal_init: hasUnifiedMemory             = true
ggml_metal_init: maxTransferRate              = built-in GPU
llama_new_context_with_model: max tensor size =   205.08 MB
ggml_metal_add_buffer: allocated 'data            ' buffer, size = 36864.00 MB, offs =            0
ggml_metal_add_buffer: allocated 'data            ' buffer, size =  8603.55 MB, offs =  38439649280, (45468.00 / 49152.00)
ggml_metal_add_buffer: allocated 'eval            ' buffer, size =    24.00 MB, (45492.00 / 49152.00)
ggml_metal_add_buffer: allocated 'kv              ' buffer, size =   642.00 MB, (46134.00 / 49152.00)
ggml_metal_add_buffer: allocated 'scr0            ' buffer, size =   456.00 MB, (46590.00 / 49152.00)
ggml_metal_add_buffer: allocated 'scr1            ' buffer, size =   304.00 MB, (46894.00 / 49152.00)

10:30:36.498 [info] Dispose Kernel process 40138.
10:30:36.498 [error] Raw kernel process exited code: undefined
10:30:36.499 [error] Error in waiting for cell to complete [Error: Canceled future for execute_request message before replies were done
	at t.KernelShellFutureHandler.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2023.6.1101941928-darwin-arm64/out/extension.node.js:2:32375)
	at ~/.vscode/extensions/ms-toolsai.jupyter-2023.6.1101941928-darwin-arm64/out/extension.node.js:2:51427
	at Map.forEach (<anonymous>)
	at v._clearKernelState (~/.vscode/extensions/ms-toolsai.jupyter-2023.6.1101941928-darwin-arm64/out/extension.node.js:2:51412)
	at v.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2023.6.1101941928-darwin-arm64/out/extension.node.js:2:44894)
	at ~/.vscode/extensions/ms-toolsai.jupyter-2023.6.1101941928-darwin-arm64/out/extension.node.js:24:113024
	at re (~/.vscode/extensions/ms-toolsai.jupyter-2023.6.1101941928-darwin-arm64/out/extension.node.js:2:1587343)
	at Cv.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2023.6.1101941928-darwin-arm64/out/extension.node.js:24:113000)
	at Ev.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2023.6.1101941928-darwin-arm64/out/extension.node.js:24:120283)
	at process.processTicksAndRejections (node:internal/process/task_queues:96:5)]
10:30:36.499 [warn] Cell completed with errors {
  message: 'Canceled future for execute_request message before replies were done'
}
10:30:36.499 [info] End cell 21 execution @ 1691602236499, started @ 1691602225152, elapsed time = 11.347s
```

Tested with llama.cpp sample and 70b model works directly without langchain. The problem only occurs when using langchain to prompt to llama.cpp & the 70b model.

### Expected behavior

Kernel should not crash.