chat_history conflict when using ConversationalRetrievalChain.from_llm
Labels: name: area: memory, name: auto:bug

### System Info

System Info:

langchain==0.0.325
Python 3.11.4
Flask==3.0.0
Flask-Cors==4.0.0
Flask-RESTful==0.3.10

Case 1:
```
dbqa = ConversationalRetrievalChain.from_llm(llm, vectordb.as_retriever(), memory=memory)
response = dbqa({"question": data["query"]})
```

Error:
Missing some input keys: {'chat_history'}

Call Stack:
`Chain.__call__()`

`Chain.prepare_inputs(inputs)` called from `__call__()` **line 286** with `inputs={'question': 'hey bot!'}`

`if self.memory is not None:` in `prepare_inputs()` passed

`external_context = self.memory.load_memory_variables(inputs)` sets `external_context` to `{'DocScanner.pdf': []}`

`self._validate_inputs(inputs)` on **line 443** 

`missing_keys = set(self.input_keys).difference(inputs)` results in `missing_keys={'chat_history'}`

Case 2:
```
dbqa = ConversationalRetrievalChain.from_llm(llm, vectordb.as_retriever(), memory=memory)
response = dbqa({"question": data["query"], "chat_history": chat_history})
```

Error:
ValueError: One input key expected got ['chat_history', 'question']

Call Stack:
`Chain.__call__()`

`Chain.prepare_inputs(inputs)` called from `__call__()` **line 286** with `inputs={'question': 'hey bot!'}`

`if self.memory is not None:` in `prepare_inputs()` passed

`external_context = self.memory.load_memory_variables(inputs)` sets `external_context` to `{'DocScanner.pdf': []}`

`self._validate_inputs(inputs)` on **line 443** 

`missing_keys = set(self.input_keys).difference(inputs)` results in `missing_keys={}`

`Chiain.__call__` line 303 sets `outputs={'answer': " Of course! I...elp today?'}`

`self.prep_outputs` called on line 312.

inside `prepare_outputs`, `if self.memory is not None:` check passes

`self.memory.save_context(inputs, outputs)` calls to `BaseChatMemory.save_context`

`input_str, output_str = self._get_input_output(inputs, outputs)` calls to `BaseChatMemory._get_input_output`

`if self.input_key is None:` check passes 

call to `langchain.memory.utils.get_prompt_input_key`

`prompt_input_keys = list(set(inputs).difference(memory_variables + ["stop"]))` returns `['chat_history', 'question']`

`if len(prompt_input_keys) != 1:` check passes raising the error


### Who can help?

Anyone

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

steps to reproduce:

1. clone [repo](https://github.com/afif-malghani/LangChain-papers)
2. create a virtual environment with `python3 -m venv env`
3. activate the virtual environment with `source env/bin/activate`
4. install all packages with `pip install -r requirements`
5. run flask server with `flask run --debug`
6. upload a pdf at the home pate of the application displayed at `http://localhost:5000/`
7. In the Chat section enter the message you want to send to the bot.

### Expected behavior

Response should be returned either with or without `chat_history`.

Currently it is not returned in either of the cases.

Edit: upgraded to langchain 0.0.325, issue persists.
Edit 2: improved readability