RetrievalQA.from_chain_type does not release GPU memory when given ChromaDB context

### System Info

When querying with no context (emptyStore below) GPU memory goes up to 8GB and after the chain completes, GPU memory goes back down to 630MB.

When using a ChromaDB to provide vector context, GPU memory is never released. Memory usage goes up to 8GB and stays there. Once enough calls have been made, the program will crash with an out of memory error.

I have tried manually deleting the variables associated with the DB and langchain, running garbage collection... I am unable to free this GPU memory. Is there a manual method to free this memory that I could employ or some other workaround?

I started with using langchain 201 and noticed the issue. The issue persists when using the latest 209.

```
def queryGpt(query):
    # Get our llm and embeddings
    llm = get_llm()
    embeddings = get_embeddings()
    # Even if the user does not specify a vector store to use, it is necessary
    # to pass in a retriever to the RetrievalQA chain.
    docs = [
        Document(page_content=""),
        Document(page_content=""),
        Document(page_content=""),
        Document(page_content=""),
    ]
    emptyStore = Chroma.from_documents(docs)
    retriever = emptyStore.as_retriever()

    if request.content_type == "application/json":
        data = request.get_json()
        store_id = data.get("store_id")
        store_collection = data.get("store_collection")
        if store_id and store_collection:
            vector_stores = load_vector_stores()
            found: VectorStore | None = None
            for store in vector_stores:
                if store["id"] == store_id:
                    found = store
            if not found:
                print(f"Warning: vector store not found id:{store_id}")
            else:
                # print(f"Using vector store '{found['name']}' id:{found['id']} collection {store_collection}")
                client = get_chroma_instance(found["dirname"])
                # embeddings = HuggingFaceEmbeddings(model_name=embeddings_model_name)
                db = Chroma(
                    client=client,
                    embedding_function=embeddings,
                    collection_name=store_collection,
                )
                retriever = db.as_retriever()
    print('Answering question')
    qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever)

    # Get the answer from the chain
    res = qa(query)
```

We are using the latest Vicuna 13b. With `all-MiniLM-L6-v2` used for the embeddings.

We are in Azure using Tesla GPU's. Ubuntu 20.04. Cuda 12.1.

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Run a QA chain with a ChromaDB enabled.

### Expected behavior

I would expect the memory to be freed upon completion of the chain.