Repository: deepset-ai/haystack
Pull request: fix: ChatPromptBuilder Fails to JSON Serialize
### Related Issues

- fixes #7844 

### Proposed Changes:

 <!--- In case of a bug: Describe what caused the issue and how you solved it -->
 <!--- In case of a feature: Describe what did you add and how it works -->
The bug was caused by the 'ChatMessage' class not being serialized correctly, which led to a failure in the serialization of 'ChatPromptTemplate'. To resolve this issue, 'to_dict' and 'from_dict' methods were added to both classes, ensuring proper serialization.
### How did you test it?

<!-- unit tests, integration tests, manual verification, instructions for manual tests -->
Added testing for all the new methods and reproduced the scenario presented on #7844 , it works fine now

### Notes for the reviewer

<!-- E.g. point out section where the reviewer  -->

### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt) ✅
- I have updated the related issue with new insights and changes ✅
- I added unit tests and updated the docstrings ✅
- I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`. ✅
- I documented my code ✅
- I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue ✅


Problem statement:
`ChatPromptBuilder` Fails to JSON Serialize due to `ChatMessage` class
**Describe the bug**
The new ChatPromptBuilder fails to serialize because of the ChatMessage dataclass. This component takes in a list of ChatMessages in the init method under the template

**Error message**
TypeError: Object of type ChatMessage is not JSON serializable

**Expected behavior**
The behavior I expect is that the component gets converted into JSON like this and that we can load with this JSON to create the pipeline back (I'm using a JSON Marshaller)

```
"components": {
    "ChatPromptBuilder": {
         "init_parameters": {
              "template": [
                   {
                        "content:" "test message"
                        "role": "user",
                        "name": "test"
                        "meta:" {
                            "test": true
                        }
                    }
              ]
         }
     }
}
```
**Additional context**
I noticed that ChatPromptBuilder and ChatMessage classes do not have any from dict or to dict functions, so it explains why they fail to serialize. In general, I've seen similar issues reported for DataClasses so I think they should all be serializable 

**To Reproduce**
```
    messages = [ChatMessage.from_system("You are a helpful assistant giving out valuable information to tourists."), ChatMessage.from_user("Tell me about {{location}}")]`
   chat_prompt_builder = ChatPromptBuilder(template=messages)
```

Hints from issue discussion:
Thanks for reporting this.

**Reproducible example**
```python
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack import Pipeline

messages = [ChatMessage.from_system("Message")]
chat_prompt_builder = ChatPromptBuilder(template=messages)

pipeline = Pipeline()
pipeline.add_component("builder", chat_prompt_builder)

pipe_str = pipeline.dumps()
new_pipeline = Pipeline().loads(pipe_str)
```
Let me try to solve this!
I been working on this for an hour now and I find that the problem here is that the 'component_to_dict' and 'deffault_to_dict' functions do not work recursively on fields that may need serialization, and just lives them as is. This makes serialization fail while we only use straightforward data types that should be serialized and deserialized without any problem.
In order to solve this specific bug we could just create the 'to_dict' and 'from_dict' methods of 'ChatPromptBuilder', but I believe that we could implement a process to serialize and deserialize instances recursively so we do not get this kind of errors again.
@CarlosFerLo go for the simple solution, custom `to_dict` and `from_dict` are the way to go.

Commit message: remove flattening parameter in to_dict

Patch to review:
diff --git a/haystack/components/builders/chat_prompt_builder.py b/haystack/components/builders/chat_prompt_builder.py
index b49509d9c9..2f36686114 100644
--- a/haystack/components/builders/chat_prompt_builder.py
+++ b/haystack/components/builders/chat_prompt_builder.py
@@ -6,7 +6,7 @@
 
 from jinja2 import Template, meta
 
-from haystack import component, logging
+from haystack import component, default_from_dict, default_to_dict, logging
 from haystack.dataclasses.chat_message import ChatMessage, ChatRole
 
 logger = logging.getLogger(__name__)
@@ -221,3 +221,36 @@ def _validate_variables(self, provided_variables: Set[str]):
                 f"Missing required input variables in ChatPromptBuilder: {missing_vars_str}. "
                 f"Required variables: {self.required_variables}. Provided variables: {provided_variables}."
             )
+
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Returns a dictionary representation of the component.
+
+        :returns:
+            Serialized dictionary representation of the component.
+        """
+        if self.template is not None:
+            template = [m.to_dict() for m in self.template]
+        else:
+            template = None
+
+        return default_to_dict(
+            self, template=template, variables=self._variables, required_variables=self._required_variables
+        )
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "ChatPromptBuilder":
+        """
+        Deserialize this component from a dictionary.
+
+        :param data:
+            The dictionary to deserialize and create the component.
+
+        :returns:
+            The deserialized component.
+        """
+        init_parameters = data["init_parameters"]
+        template = init_parameters.get("template", [])
+        init_parameters["template"] = [ChatMessage.from_dict(d) for d in template]
+
+        return default_from_dict(cls, data)
diff --git a/haystack/dataclasses/chat_message.py b/haystack/dataclasses/chat_message.py
index 336f5a654f..d8e7de8e43 100644
--- a/haystack/dataclasses/chat_message.py
+++ b/haystack/dataclasses/chat_message.py
@@ -2,7 +2,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-from dataclasses import dataclass, field
+from dataclasses import asdict, dataclass, field, fields
 from enum import Enum
 from typing import Any, Dict, Optional
 
@@ -99,3 +99,51 @@ def from_function(cls, content: str, name: str) -> "ChatMessage":
         :returns: A new ChatMessage instance.
         """
         return cls(content, ChatRole.FUNCTION, name)
+
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Converts ChatMessage into a dictionary.
+
+        :returns:
+            Serialized version of the object.
+        """
+        data = asdict(self)
+        data["role"] = self.role.value
+
+        meta = data.pop("meta")
+        return {**data, **meta}
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "ChatMessage":
+        """
+        Creates a new ChatMessage object from a dictionary.
+
+        :param data:
+            The dictionary to build the ChatMessage object.
+        :returns:
+            The created object.
+        """
+        data["role"] = ChatRole(data["role"])
+
+        # Store metadata for a moment while we try un-flattening allegedly flatten metadata.
+        # We don't expect both a `meta=` keyword and flatten metadata keys so we'll raise a
+        # ValueError later if this is the case.
+        meta = data.pop("meta", {})
+        # Unflatten metadata if it was flattened. We assume any keyword argument that's not
+        # a document field is a metadata key. We treat legacy fields as document fields
+        # for backward compatibility.
+        flatten_meta = {}
+        document_fields = [f.name for f in fields(cls)]
+        for key in list(data.keys()):
+            if key not in document_fields:
+                flatten_meta[key] = data.pop(key)
+
+        # We don't support passing both flatten keys and the `meta` keyword parameter
+        if meta and flatten_meta:
+            raise ValueError(
+                "You can pass either the 'meta' parameter or flattened metadata keys as keyword arguments, "
+                "but currently you're passing both. Pass either the 'meta' parameter or flattened metadata keys."
+            )
+
+        # Finally put back all the metadata
+        return cls(**data, meta={**meta, **flatten_meta})
diff --git a/releasenotes/notes/fix-chat-prompt-builder-serialization-345605337ca2584e.yaml b/releasenotes/notes/fix-chat-prompt-builder-serialization-345605337ca2584e.yaml
new file mode 100644
index 0000000000..b9df38134d
--- /dev/null
+++ b/releasenotes/notes/fix-chat-prompt-builder-serialization-345605337ca2584e.yaml
@@ -0,0 +1,4 @@
+---
+fixes:
+  - |
+    Solve serialization bug on 'ChatPromptBuilder' by creating 'to_dict' and 'from_dict' methods on 'ChatMessage' and 'ChatPromptBuilder'.
diff --git a/test/components/builders/test_chat_prompt_builder.py b/test/components/builders/test_chat_prompt_builder.py
index 406e40b739..bbcaedf7bd 100644
--- a/test/components/builders/test_chat_prompt_builder.py
+++ b/test/components/builders/test_chat_prompt_builder.py
@@ -494,3 +494,44 @@ def run(self):
                 ]
             }
         }
+
+    def test_to_dict(self):
+        component = ChatPromptBuilder(
+            template=[ChatMessage.from_user("text and {var}"), ChatMessage.from_assistant("content {required_var}")],
+            variables=["var", "required_var"],
+            required_variables=["required_var"],
+        )
+
+        assert component.to_dict() == {
+            "type": "haystack.components.builders.chat_prompt_builder.ChatPromptBuilder",
+            "init_parameters": {
+                "template": [
+                    {"content": "text and {var}", "role": "user", "name": None},
+                    {"content": "content {required_var}", "role": "assistant", "name": None},
+                ],
+                "variables": ["var", "required_var"],
+                "required_variables": ["required_var"],
+            },
+        }
+
+    def test_from_dict(self):
+        component = ChatPromptBuilder.from_dict(
+            data={
+                "type": "haystack.components.builders.chat_prompt_builder.ChatPromptBuilder",
+                "init_parameters": {
+                    "template": [
+                        {"content": "text and {var}", "role": "user", "name": None},
+                        {"content": "content {required_var}", "role": "assistant", "name": None},
+                    ],
+                    "variables": ["var", "required_var"],
+                    "required_variables": ["required_var"],
+                },
+            }
+        )
+
+        assert component.template == [
+            ChatMessage.from_user("text and {var}"),
+            ChatMessage.from_assistant("content {required_var}"),
+        ]
+        assert component._variables == ["var", "required_var"]
+        assert component._required_variables == ["required_var"]
diff --git a/test/dataclasses/test_chat_message.py b/test/dataclasses/test_chat_message.py
index ba3b40533e..8cbae1121e 100644
--- a/test/dataclasses/test_chat_message.py
+++ b/test/dataclasses/test_chat_message.py
@@ -81,3 +81,35 @@ def test_apply_custom_chat_templating_on_chat_message():
         formatted_messages, chat_template=anthropic_template, tokenize=False
     )
     assert tokenized_messages == "You are good assistant\nHuman: I have a question\nAssistant:"
+
+
+def test_to_dict_with_flattening():
+    message = ChatMessage.from_user("content")
+    message.meta["some"] = "some"
+
+    assert message.to_dict() == {"content": "content", "role": "user", "name": None, "some": "some"}
+
+
+def test_from_dict():
+    assert ChatMessage.from_dict(data={"content": "text", "role": "user", "name": None}) == ChatMessage(
+        content="text", role=ChatRole("user"), name=None, meta={}
+    )
+
+
+def test_from_dict_with_flat_meta():
+    assert ChatMessage.from_dict(
+        data={"content": "text", "role": "user", "name": None, "something": "something"}
+    ) == ChatMessage(content="text", role=ChatRole("user"), name=None, meta={"something": "something"})
+
+
+def test_from_dict_with_non_flat_meta():
+    assert ChatMessage.from_dict(
+        data={"content": "text", "role": "user", "name": None, "meta": {"something": "something"}}
+    ) == ChatMessage(content="text", role=ChatRole("user"), name=None, meta={"something": "something"})
+
+
+def test_from_dict_with_flat_and_non_flat_meta():
+    with pytest.raises(ValueError):
+        ChatMessage.from_dict(
+            data={"content": "content", "role": "user", "name": None, "meta": {"field": "something"}, "more": "fields"}
+        )


Code review comments:
diff_hunk: @@ -99,3 +99,51 @@ def from_function(cls, content: str, name: str) -> "ChatMessage":
         :returns: A new ChatMessage instance.
         """
         return cls(content, ChatRole.FUNCTION, name)
+
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Converts ChatMessage into a dictionary.
+
+        :returns:
+            Serialized version of the object.
+        """
+        data = asdict(self)
+        data["role"] = self.role.value
+
+        meta = data.pop("meta")
+        return {**data, **meta}
path: haystack/dataclasses/chat_message.py
text: @user1:
The metadata could contain a key with the same name as one of the dataclass' field. We should keep it as a nested dict.

@author:
True, I did not think about it. I believed that as it was the default in documents, we could just keep the flattened standard here.

@user1:
Let's keep it nested.

@author:
okey

diff_hunk: @@ -99,3 +99,51 @@ def from_function(cls, content: str, name: str) -> "ChatMessage":
         :returns: A new ChatMessage instance.
         """
         return cls(content, ChatRole.FUNCTION, name)
+
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Converts ChatMessage into a dictionary.
+
+        :returns:
+            Serialized version of the object.
+        """
+        data = asdict(self)
+        data["role"] = self.role.value
+
+        meta = data.pop("meta")
+        return {**data, **meta}
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "ChatMessage":
+        """
+        Creates a new ChatMessage object from a dictionary.
+
+        :param data:
+            The dictionary to build the ChatMessage object.
+        :returns:
+            The created object.
+        """
+        data["role"] = ChatRole(data["role"])
+
+        # Store metadata for a moment while we try un-flattening allegedly flatten metadata.
+        # We don't expect both a `meta=` keyword and flatten metadata keys so we'll raise a
+        # ValueError later if this is the case.
+        meta = data.pop("meta", {})
+        # Unflatten metadata if it was flattened. We assume any keyword argument that's not
+        # a document field is a metadata key. We treat legacy fields as document fields
+        # for backward compatibility.
+        flatten_meta = {}
+        document_fields = [f.name for f in fields(cls)]
+        for key in list(data.keys()):
+            if key not in document_fields:
+                flatten_meta[key] = data.pop(key)
+
+        # We don't support passing both flatten keys and the `meta` keyword parameter
+        if meta and flatten_meta:
+            raise ValueError(
+                "You can pass either the 'meta' parameter or flattened metadata keys as keyword arguments, "
+                "but currently you're passing both. Pass either the 'meta' parameter or flattened metadata keys."
+            )
path: haystack/dataclasses/chat_message.py
text: @user1:
Looks like this code is redundant since we no longer support flattening?

@author:
I believe we could still support for not standard deserialisation of 'ChatMessage' as we could use it sometime, as it keeps the input dictionary more readable. For example, someone might want to create a pipeline where a generator creates dictionaries containing chat messages, and you do not want to make the LLM have to write excessively complex dictionaries as it will be more prone to fail. But if you feel that it makes no sense to support non-standard serialization, we could just delete the code.

@user1:
Let's keep this bugfix PR as simple as possible by just wrapping `dataclasses.asdict()` and `ChatMessage(**serialized_dict)`.

@author:
okey