Multiple GPU - Degradation over time
Labels: name: auto:bug, name: area: models

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

```llm = LlamaCpp(model_path=model_loc, n_ctx=0, verbose=True, n_gpu_layers=-1)
chain = LLMChain(llm=llm, prompt=prompt)
result = chain(inputs={"context": context, "question": question})
answer = result["text"]
```
I do use the PromptTemplate, but this is not relevant to my issue.

### Description

I am using 2x A100 GPU.  What I have noticed is that there is GPU degradation over time.

This is my spool up data:

```
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 2 CUDA devices:
  Device 0: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: no
  Device 1: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: no
```

In my code, the above is part of an API that takes in a request and then responds to it.  I am making 5 calls to the API one after another, and collating the data for a final summarisation call.

I am not overly impressed with the speeds here in any event - so if anyone has any tips for improving, I don't think 17.25 tokens is very good for dual A100s.

```
llama_print_timings:        eval time =      57.98 ms /     1 runs   (   57.98 ms per token,    17.25 tokens per second)
llama_print_timings:       total time =    6722.12 ms

llama_print_timings:        eval time =    4804.66 ms /    70 runs   (   68.64 ms per token,    14.57 tokens per second)
llama_print_timings:       total time =   17722.48 ms

llama_print_timings:        eval time =    5950.61 ms /    77 runs   (   77.28 ms per token,    12.94 tokens per second)
llama_print_timings:       total time =   23459.58 ms
```

I would have expected the "overall" time to change but the actual tokens per second to remain somewhat consistent - but instead it just gets worse.

I also created a GPU monitor tool, and in that tool, the first call is utilising 25% and 20% for GPU 1 and 2 - but then it goes down to 20% and 17% and then 16% and 15%, and when making a LOT of calls, I have seen it as low as 9% on each GPU - at this point the response time is a minute plus.

Why am I writing a bug report here? Sounds like a you problem....because I ran it with Llama-cpp-python - without Langchain and this was not a problem, there was no degradation over time.


### System Info

Wrinkles....

I am using Window 10, inside a VMWare Virtual Machine.  Python, Langchain, Llama-cpp-python is all up to date, as are the Nvidia drivers and CUDA toolkit (12.2).

As noted above - this only occurs when using langchain - not when doing the same workflow, without langchain.

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async