Repository: langchain-ai/langchain (Python)
Date: 2023-05-14 01:54:58+00:00
Task type: Bug Fix

Issue:
PydanticOutputParser has high chance failing when completion contains new line
## Context
When the completion is of a longer format such as an Email, the text will likely contain new line character `\n`.
If it is not properly escaped like `\\n`, parsing will fail when using PydanticOutputParser as `json.loads` does not allow control characters in strict mode.
Most of the time, RetryWithErrorOutputParser also fails to correct the format.

## Example
```python
from langchain.output_parsers import PydanticOutputParser
from langchain.prompts import PromptTemplate
from pydantic import BaseModel, Field

class Email(BaseModel):
    subject: str = Field(description="main objective of the email")
    body: str = Field(description="email content")

parser = PydanticOutputParser(pydantic_object=Email)

prompt = PromptTemplate(
    template="Answer the user query.\n{format_instructions}\n{query}\n",
    input_variables=["query"],
    partial_variables={"format_instructions": parser.get_format_instructions()},
)
completion = llm(
    prompt.format(
        query="Write a long formal email to inform my clients that the company is broke."
    )
)

parser.parse(completion)
```
```python
# completion

> Here is the output instance:
\```
{"subject": "Company Status Update", "body": "Dear Clients, 

This email is to inform you that our company is currently in a difficult financial situation. We apologize for any inconvenience caused by this and are doing our best to ensure that our services remain of the highest quality for our valued clients.

We want to thank you for your support and understanding during this difficult time.

Sincerely,
[Company Name]"}
\```
```
```python
# parser.parse(completion)

> Got: Invalid control character at: line 1 column 61 (char 60)
```

## Thoughts
Maybe include instructions on escaping in PYDANTIC_FORMAT_INSTRUCTIONS?

Or could adding an option to allow non-strict mode be considered?
https://github.com/hwchase17/langchain/blob/32793f94fd6da0bb36311e1af4051f7883dd12c5/langchain/output_parsers/pydantic.py#L25


Issue discussion:

Solution patch:
diff --git a/langchain/output_parsers/pydantic.py b/langchain/output_parsers/pydantic.py
--- a/langchain/output_parsers/pydantic.py
+++ b/langchain/output_parsers/pydantic.py
@@ -22,7 +22,7 @@ def parse(self, text: str) -> T:
             json_str = ""
             if match:
                 json_str = match.group()
-            json_object = json.loads(json_str)
+            json_object = json.loads(json_str, strict=False)
             return self.pydantic_object.parse_obj(json_object)
 
         except (json.JSONDecodeError, ValidationError) as e:


Test patch:
diff --git a/tests/unit_tests/output_parsers/test_pydantic_parser.py b/tests/unit_tests/output_parsers/test_pydantic_parser.py
--- a/tests/unit_tests/output_parsers/test_pydantic_parser.py
+++ b/tests/unit_tests/output_parsers/test_pydantic_parser.py
@@ -21,6 +21,7 @@ class TestModel(BaseModel):
     additional_fields: Optional[str] = Field(
         description="Additional fields", default=None
     )
+    for_new_lines: str = Field(description="To be used to test newlines")
 
 
 # Prevent pytest from trying to run tests on TestModel
@@ -30,7 +31,8 @@ class TestModel(BaseModel):
 DEF_RESULT = """{
     "action": "Update",
     "action_input": "The PydanticOutputParser class is powerful",
-    "additional_fields": null
+    "additional_fields": null,
+    "for_new_lines": "not_escape_newline:\n escape_newline: \\n"
 }"""
 
 # action 'update' with a lowercase 'u' to test schema validation failure.
@@ -44,6 +46,7 @@ class TestModel(BaseModel):
     action=Actions.UPDATE,
     action_input="The PydanticOutputParser class is powerful",
     additional_fields=None,
+    for_new_lines="not_escape_newline:\n escape_newline: \n",
 )