QAGenerationChain.run often ends in JSONDecodeError
Labels: name: auto:bug, name: auto:improvement, name: area: models

The `QAGenerationChain` as it is currently written is prone to a `JSONDecodeError`, as mentioned in https://github.com/langchain-ai/langchain/pull/9503. That was my naive attempt to fix the problem I was having, but as I explained in closing the PR, I think a `PydanticOutputParser` with a more instructive prompt or an auto-fixing parser would be more robust. Plus, I think the successful runs after implementing my fix were just luck. 🤣 

In `QAGenerationChain._call`, after generation, `json.loads` frequently raises a `JSONDecodeError`. This is usually because the response is wrapped in Markdown code tags or prefaced with a message, as seen in the following traces:

- `ChatOpenAI` gpt-3.5-turbo: [successful trace](https://smith.langchain.com/public/6c42845c-bbe2-41c1-99a4-7669112f1504/r)
- `ChatOpenAI` gpt-4: [`JSONDecodeError` due to Markdown formatting](https://smith.langchain.com/public/d235ef1e-fb13-4177-adb3-6ddaf159bfea/r)
- `ChatAnthropic` Claude 2: [`JSONDecodeError` due to message before the JSON + Markdown formatting](https://smith.langchain.com/public/27eed20e-9ca5-4b4a-a3d3-71efde7aee3c/r)
- `ChatAnyscale` llama 2 70b: [successful trace](https://smith.langchain.com/public/e8f0b44c-424f-47ef-8cb3-31713b548d46/r?tab=0)
- `ChatAnyscale` llama 2 70b: [`JSONDecodeError` due to message before the JSON + Markdown formatting](https://smith.langchain.com/public/c89a7415-477d-4b82-a0a4-0a333b8b5680/r)

Each of these models had a mix of successful and failed runs. I probably should have lowered the temperature and tried non-chat models as well.


![image](https://github.com/langchain-ai/langchain/assets/84336755/1e4da267-3b7e-40c0-b726-3a0304533b44)


I would like to help however I could on this, I just want to be sure my fix aligns with the greater vision for this chain since it seems particularly useful. :)

### System Info

Version info from LangSmith:
```
RUNTIME
langchain_version: "0.0.268"
library: "langchain"
library_version: "0.0.268"
platform: "Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.36"
runtime: "python"
runtime_version: "3.11.4"
sdk_version: "0.0.25"
```

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
from langchain.chains import QAGenerationChain

# tested with gpt-3.5-turbo, gpt-4, claude 2, and llama 2 70b

qagen = QAGenerationChain.from_llm(llm=llm)

question = qagen.run(docs[0].page_content)

# JSONDecodeError (sometimes)
```

The error is inconsistent, as seen in the screenshot above. This is for reasons we're all familiar with: perhaps the output is wrapped in Markdown code formatting with triple-backticks, maybe it's prefaced with a "sure, here's your JSON", etc.

Relevant examples from LangSmith:

- `ChatOpenAI` gpt-3.5-turbo: [successful trace](https://smith.langchain.com/public/6c42845c-bbe2-41c1-99a4-7669112f1504/r)
- `ChatOpenAI` gpt-4: [`JSONDecodeError` due to Markdown formatting](https://smith.langchain.com/public/d235ef1e-fb13-4177-adb3-6ddaf159bfea/r)
- `ChatAnthropic` Claude 2: [`JSONDecodeError` due to message before the JSON + Markdown formatting](https://smith.langchain.com/public/27eed20e-9ca5-4b4a-a3d3-71efde7aee3c/r)
- `ChatAnyscale` llama 2 70b: [successful trace](https://smith.langchain.com/public/e8f0b44c-424f-47ef-8cb3-31713b548d46/r?tab=0)
- `ChatAnyscale` llama 2 70b: [`JSONDecodeError` due to message before the JSON + Markdown formatting](https://smith.langchain.com/public/c89a7415-477d-4b82-a0a4-0a333b8b5680/r)


### Expected behavior

`QAGenerationChain.run` should return its expected output with no `JSONDecodeError`. I realize there's only so much we can do for the chain's default settings, and an LLM call can be inherently predictable, but some parsing and error handling would be nice. :)