Chain Type Refine Error: 1 validation error for RefineDocumentsChain prompt  extra fields not permitted 
Labels: name: area: memory, name: auto:bug, name: area: models, name: integration: chroma

### System Info

langchain                 0.0.326 

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain.chains import RetrievalQA
from langchain.memory import ConversationBufferMemory
from langchain.prompts import PromptTemplate

# This text splitter is used to create the parent documents - The big chunks
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=400)

# This text splitter is used to create the child documents - The small chunks
# It should create documents smaller than the parent
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)

# The vectorstore to use to index the child chunks
from chromadb.errors import InvalidDimensionException
try:
    vectorstore = Chroma(collection_name="split_parents", embedding_function=bge_embeddings, persist_directory="chroma_db")
except InvalidDimensionException:
    Chroma().delete_collection()
    vectorstore = Chroma(collection_name="split_parents", embedding_function=bge_embeddings, persist_directory="chroma_db")

#vectorstore = Chroma(collection_name="split_parents", embedding_function=bge_embeddings) 

# The storage layer for the parent documents
store = InMemoryStore()

big_chunks_retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

big_chunks_retriever.add_documents(documents)

qa_template = """
Nutze die folgenden Informationen aus dem Kontext (getrennt mit <ctx></ctx>), um die Frage zu beantworten.
Antworte nur auf Deutsch, weil der Nutzer kein Englisch versteht! \
Falls du die Antwort nicht weißt, antworte mit "Leider fehlen mir dazu die Informationen." \
Wenn du nicht genügend Informationen unten findest, antworte ebenfalls mit "Leider fehlen mir dazu die Informationen." \
------
<ctx>
{context}
</ctx>
------
{query}
Answer:
"""

prompt = PromptTemplate(template=qa_template,
                            input_variables=['context','history', 'question'])

chain_type_kwargs={
        "verbose": True,
        "prompt": prompt,
        "memory": ConversationSummaryMemory(
            llm=build_llm(),
            memory_key="history",
            input_key="question",
            return_messages=True)}

refine = RetrievalQA.from_chain_type(llm=build_llm(),
                                 chain_type="refine",
                                 return_source_documents=True,
                                 chain_type_kwargs=chain_type_kwargs,
                                 retriever=big_chunks_retriever,
                                 verbose=True)

query = "Hi, I am Max, can you help me??"
refine(query)
```

### Expected behavior

Hi, 

in the code you see how I built my RAG model with the ParentDocumentRetriever from Langchain and with Memory. At the moment I am using the RetrievalQA-Chain with the default  `chain_type="stuff"`. However I want to try different chain types like "map_reduce" or "refine". But when replacing `chain_type="refine"` and creating the Retrieval QA chain, I get the following Error: 
```
ValidationError: 1 validation error for RefineDocumentsChain
prompt
  extra fields not permitted (type=value_error.extra)
```

How can I solve this?