Repository: tefra/xsdata (python)

Issue:
Convert XML sample into cannonical python representation code
# Motivation
We maintain set of XML samples conforming to specific XML schema.

We are considering to generate these samples by python code using xsdata and xsdata generated package.

It would be helpful to have an option to generate the python representation by converting XML samples into it.

# Context
The python package allowing to build XML objects depends on XML schema and on backing object library:
- Default backing library is based on dataclasses.
- Another one is based on attrs lib (if you install xsdata-attrs).
- In future we may expect pydantic based or others.

If we load an XML sample into xsdata based objects, we may:
- serialize it to XML
- serialize it to JSON
- use repr or str to print something close to Python code

# Proposed approach - pycode serializer
Currently, we use XmlSerializer
```python
from xsdata.formats.dataclass.serializers import XmlSerializer
from xsdata.formats.dataclass.serializers.config import SerializerConfig
```
If the backing library provides PycodeSerializer, we can get the python code generated.

Such a serializer would be optional (so new backing libraries would not be obliged to provide that).

To get the python code, one would find in documentation example similar to https://xsdata.readthedocs.io/en/latest/json.html#serialize-json-to-string or we could even add section Python code binding, which would not only serialize.

The code would be similar to:
```python
from tests.fixtures.books.books import *
from xsdata.models.datatype import XmlDate

from xsdata.formats.dataclass.context import XmlContext
from xsdata.formats.dataclass.parsers import XmlParser

from xsdata.formats.dataclass.serializers import PycodeSerializer
from xsdata.formats.dataclass.serializers.config import SerializerConfig


xmlpath = Path("books.xml")
xml_str = xmlpath.read_text(encoding="utf-8")

parser = XmlParser(context=XmlContext())
books = parser.from_string(xml_str, Book)

config = SerializerConfig()
serializer = PycodeSerializer(context=XmlContext(), config=config)

pycode_str = serializer.render(books)

print(pycode_str)
```
and the generated python code would be:

```python
from tests.fixtures.books.books import *
from xsdata.models.datatype import XmlDate

Books(
   book=[
       BookForm(
           id="bk001",
           author="Hightower, Kim",
           title="The First Book",
           genre="Fiction",
           price=44.95,
           review="An amazing story of nothing.",
       ),
       BookForm(
           id="bk002",
           author="Nagata, Suanne",
           title="Becoming Somebody",
           price=33.95,
           pub_date=XmlDate(2001, 1, 10),
           review="A masterpiece of the fine art of gossiping.",
       ),
   ]
)

Even better support would be using CLI (but it is not really necessary).
```


Pull request:
Add pycode serializer
## 📒 Description

Add a serializer that renders a python representation code for a dataclass.
This will help me alot with visualize without debugging what is really happending during parsing and how the result object is constructed.


Resolves #626 

## 🔗 What I've Done


## 💬 Comments

## 🛫 Checklist

- [x] Updated docs
- [x] Added unit-tests
- [x] [Sample tests](https://github.com/tefra/xsdata-samples) pass
- [x] [W3C tests](https://github.com/tefra/xsdata-w3c-tests) pass


Patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index d227e854..ddf1a1e4 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -2,12 +2,12 @@ exclude: tests/fixtures
 
 repos:
   - repo: https://github.com/asottile/pyupgrade
-    rev: v2.37.1
+    rev: v2.37.3
     hooks:
       - id: pyupgrade
         args: [--py37-plus]
   - repo: https://github.com/asottile/reorder_python_imports
-    rev: v3.8.1
+    rev: v3.8.2
     hooks:
       - id: reorder-python-imports
   - repo: https://github.com/ambv/black
@@ -37,16 +37,16 @@ repos:
       - id: docformatter
         args: ["--in-place", "--pre-summary-newline"]
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v0.961
+    rev: v0.971
     hooks:
       - id: mypy
         additional_dependencies: [tokenize-rt, types-requests, types-Jinja2, types-click, types-docutils]
   - repo: https://github.com/asottile/setup-cfg-fmt
-    rev: v1.20.1
+    rev: v2.0.0
     hooks:
       - id: setup-cfg-fmt
-        args: ["--max-py-version=3.11"]
+        args: ["--max-py-version=3.11", "--include-version-classifiers"]
   - repo: https://github.com/PyCQA/doc8
-    rev: 0.11.2
+    rev: v1.0.0
     hooks:
       - id: doc8
diff --git a/docs/api/serializing.rst b/docs/api/serializing.rst
index 0d86fe8e..b9ca4250 100644
--- a/docs/api/serializing.rst
+++ b/docs/api/serializing.rst
@@ -11,6 +11,7 @@ Serializing
 
     XmlSerializer
     JsonSerializer
+    PycodeSerializer
 
 .. currentmodule:: xsdata.formats.dataclass.serializers.config
 
diff --git a/docs/data-types.rst b/docs/data-types.rst
index 66ccc2cf..5aa6c9d8 100644
--- a/docs/data-types.rst
+++ b/docs/data-types.rst
@@ -82,9 +82,9 @@ Use the field metadata to set the target `format` according to the
     ...
     >>> @dataclass
     ... class Dates:
-    ...     date: dt.date = field(metadata=dict(format="%d %B %Y"))
+    ...     date: dt.date = field(metadata=dict(format="%d/%m/%Y"))
     ...     time: dt.time = field(metadata=dict(format="%H:%M"))
-    ...     datetime: dt.datetime = field(metadata=dict(format="%d %B %Y %H:%M"))
+    ...     datetime: dt.datetime = field(metadata=dict(format="%d/%m/%Y %H:%M"))
     ...
     >>> obj = Dates(
     ...     date=dt.date(2020, 1, 6),
@@ -97,9 +97,9 @@ Use the field metadata to set the target `format` according to the
     >>> print(serializer.render(obj))
     <?xml version="1.0" encoding="UTF-8"?>
     <Dates>
-      <date>06 January 2020</date>
+      <date>06/01/2020</date>
       <time>18:15</time>
-      <datetime>06 January 2020 18:15</datetime>
+      <datetime>06/01/2020 18:15</datetime>
     </Dates>
     <BLANKLINE>
 
diff --git a/docs/examples.rst b/docs/examples.rst
index a688051c..86c4102f 100644
--- a/docs/examples.rst
+++ b/docs/examples.rst
@@ -11,6 +11,7 @@ Code Generation
     examples/docstrings
     examples/xml-modeling
     examples/json-modeling
+    examples/pycode-serializer
     examples/dtd-modeling
     examples/compound-fields
     examples/dataclasses-features
diff --git a/docs/examples/dataclasses-features.rst b/docs/examples/dataclasses-features.rst
index e8fb1956..bf55fc1a 100644
--- a/docs/examples/dataclasses-features.rst
+++ b/docs/examples/dataclasses-features.rst
@@ -31,9 +31,10 @@ toggle all of them through the cli flags or a :ref:`generator config <Generator
         xml_path = fixtures_dir.joinpath("stripe/samples/balance.json")
         parser = JsonParser()
         root = parser.from_path(xml_path, Balance)
-        pprint.pprint(root.pending)
+        print(root.pending[0])
+        print(root.pending[1])
 
     .. testoutput::
 
-        (Pending(amount=835408472, currency='usd', source_types=SourceTypes(bank_account=0, card=835408472)),
-         Pending(amount=-22251, currency='eur', source_types=SourceTypes(bank_account=0, card=-22251)))
+        Pending(amount=835408472, currency='usd', source_types=SourceTypes(bank_account=0, card=835408472))
+        Pending(amount=-22251, currency='eur', source_types=SourceTypes(bank_account=0, card=-22251))
diff --git a/docs/examples/pycode-serializer.rst b/docs/examples/pycode-serializer.rst
new file mode 100644
index 00000000..2b8c9152
--- /dev/null
+++ b/docs/examples/pycode-serializer.rst
@@ -0,0 +1,49 @@
+=================
+Pycode Serializer
+=================
+
+The pycode serializer will render an object tree into
+python representation code.
+
+
+.. testcode::
+
+    from tests.fixtures.books.fixtures import books
+    from xsdata.formats.dataclass.serializers import PycodeSerializer
+    from xsdata.models.enums import Namespace
+
+    serializer = PycodeSerializer()
+    print(serializer.render(books, var_name="books"))
+
+.. testoutput::
+    :options: +NORMALIZE_WHITESPACE
+
+    from tests.fixtures.books.books import BookForm
+    from tests.fixtures.books.books import Books
+    from xsdata.models.datatype import XmlDate
+
+
+    books = Books(
+        book=[
+            BookForm(
+                author="Hightower, Kim",
+                title="The First Book",
+                genre="Fiction",
+                price=44.95,
+                pub_date=XmlDate(2000, 10, 1),
+                review="An amazing story of nothing.",
+                id="bk001",
+                lang="en"
+            ),
+            BookForm(
+                author="Nagata, Suanne",
+                title="Becoming Somebody",
+                genre="Biography",
+                price=33.95,
+                pub_date=XmlDate(2001, 1, 10),
+                review="A masterpiece of the fine art of gossiping.",
+                id="bk002",
+                lang="en"
+            ),
+        ]
+    )
diff --git a/script.py b/script.py
new file mode 100644
index 00000000..53d70641
--- /dev/null
+++ b/script.py
@@ -0,0 +1,5 @@
+from tests.fixtures.books.fixtures import books
+from xsdata.formats.dataclass.serializers.code import PycodeSerializer
+
+serializer = PycodeSerializer()
+print(serializer.render(books))
diff --git a/xsdata/formats/dataclass/filters.py b/xsdata/formats/dataclass/filters.py
index 302bf136..80d97504 100644
--- a/xsdata/formats/dataclass/filters.py
+++ b/xsdata/formats/dataclass/filters.py
@@ -1,4 +1,3 @@
-import math
 import re
 import textwrap
 from collections import defaultdict
@@ -10,8 +9,6 @@ from typing import List
 from typing import Optional
 from typing import Tuple
 from typing import Type
-from xml.etree.ElementTree import QName
-from xml.sax.saxutils import quoteattr
 
 from docformatter import format_code
 from jinja2 import Environment
@@ -27,6 +24,7 @@ from xsdata.models.config import OutputFormat
 from xsdata.utils import collections
 from xsdata.utils import namespaces
 from xsdata.utils import text
+from xsdata.utils.objects import literal_value
 
 
 class Filters:
@@ -383,7 +381,7 @@ class Filters:
         if isinstance(data, str):
             return self.format_string(data, indent, key, 4)
 
-        return self.literal_value(data)
+        return literal_value(data)
 
     def format_dict(self, data: Dict, indent: int) -> str:
         """Return a pretty string representation of a dict."""
@@ -530,7 +528,7 @@ class Filters:
         if attr.default is None:
             return False if self.format.kw_only and not attr.is_optional else None
         if not isinstance(attr.default, str):
-            return self.literal_value(attr.default)
+            return literal_value(attr.default)
         if attr.default.startswith("@enum@"):
             return self.field_default_enum(attr)
 
@@ -539,7 +537,7 @@ class Filters:
         if attr.is_tokens:
             return self.field_default_tokens(attr, types, ns_map)
 
-        return self.literal_value(
+        return literal_value(
             converter.deserialize(
                 attr.default, types, ns_map=ns_map, format=attr.restrictions.format
             )
@@ -662,19 +660,6 @@ class Filters:
 
         return self.type_name(attr_type)
 
-    @classmethod
-    def literal_value(cls, value: Any) -> str:
-        if isinstance(value, str):
-            return quoteattr(value)
-
-        if isinstance(value, float):
-            return str(value) if math.isfinite(value) else f'float("{value}")'
-
-        if isinstance(value, QName):
-            return f'QName("{value.text}")'
-
-        return repr(value).replace("'", '"')
-
     def default_imports(self, output: str) -> str:
         """Generate the default imports for the given package output."""
         result = []
diff --git a/xsdata/formats/dataclass/serializers/__init__.py b/xsdata/formats/dataclass/serializers/__init__.py
index 86f74f69..55ba1f1f 100644
--- a/xsdata/formats/dataclass/serializers/__init__.py
+++ b/xsdata/formats/dataclass/serializers/__init__.py
@@ -1,5 +1,11 @@
+from xsdata.formats.dataclass.serializers.code import PycodeSerializer
 from xsdata.formats.dataclass.serializers.json import DictFactory
 from xsdata.formats.dataclass.serializers.json import JsonSerializer
 from xsdata.formats.dataclass.serializers.xml import XmlSerializer
 
-__all__ = ["JsonSerializer", "DictFactory", "XmlSerializer"]
+__all__ = [
+    "DictFactory",
+    "JsonSerializer",
+    "XmlSerializer",
+    "PycodeSerializer",
+]
diff --git a/xsdata/formats/dataclass/serializers/code.py b/xsdata/formats/dataclass/serializers/code.py
new file mode 100644
index 00000000..f2bfaa1b
--- /dev/null
+++ b/xsdata/formats/dataclass/serializers/code.py
@@ -0,0 +1,137 @@
+from dataclasses import dataclass
+from dataclasses import field
+from enum import Enum
+from io import StringIO
+from typing import Any
+from typing import List
+from typing import Mapping
+from typing import Set
+from typing import TextIO
+from typing import Type
+
+from xsdata.formats.bindings import AbstractSerializer
+from xsdata.formats.dataclass.context import XmlContext
+from xsdata.formats.dataclass.serializers.config import SerializerConfig
+from xsdata.utils import collections
+from xsdata.utils.objects import literal_value
+
+spaces = "    "
+
+
+@dataclass
+class PycodeSerializer(AbstractSerializer):
+    """
+    Pycode serializer for dataclasses.
+
+    Return a python representation code of
+    a model instance.
+
+    :param config: Serializer configuration
+    :param context: Model context provider
+    """
+
+    config: SerializerConfig = field(default_factory=SerializerConfig)
+    context: XmlContext = field(default_factory=XmlContext)
+
+    def render(self, obj: object, var_name: str = "obj") -> str:
+        """
+        Convert and return the given object tree as python representation code.
+
+        :param obj: The input dataclass instance
+        :param var_name: The var name to assign the model instance
+        """
+        output = StringIO()
+        self.write(output, obj, var_name)
+        return output.getvalue()
+
+    def write(self, out: TextIO, obj: Any, var_name: str):
+        """
+        Write the given object tree to the output text stream.
+
+        :param out: The output stream
+        :param obj: The input dataclass instance
+        :param var_name: The var name to assign the model instance
+        """
+        types: Set[Type] = set()
+
+        tmp = StringIO()
+        for chunk in self.write_object(obj, 0, types):
+            tmp.write(chunk)
+
+        imports = self.build_imports(types)
+        out.write(imports)
+        out.write("\n\n")
+        out.write(f"{var_name} = ")
+        out.write(tmp.getvalue())
+        out.write("\n")
+
+    @classmethod
+    def build_imports(cls, types: Set[Type]) -> str:
+        imports = []
+        for tp in types:
+            module = tp.__module__
+            name = tp.__qualname__
+            if module == "builtins" or "." in name:
+                continue
+
+            imports.append(f"from {module} import {name}\n")
+
+        return "".join(sorted(imports))
+
+    def write_object(self, obj: Any, level: int, types: Set[Type]):
+        types.add(type(obj))
+        if collections.is_array(obj):
+            yield from self.write_array(obj, level, types)
+        elif isinstance(obj, dict):
+            yield from self.write_mapping(obj, level, types)
+        elif self.context.class_type.is_model(obj):
+            yield from self.write_class(obj, level, types)
+        elif isinstance(obj, Enum):
+            yield str(obj)
+        else:
+            yield literal_value(obj)
+
+    def write_array(self, obj: List, level: int, types: Set[Type]):
+        if not obj:
+            yield str(obj)
+            return
+
+        next_level = level + 1
+        yield "[\n"
+        for val in obj:
+            yield spaces * next_level
+            yield from self.write_object(val, next_level, types)
+            yield ",\n"
+
+        yield f"{spaces * level}]"
+
+    def write_mapping(self, obj: Mapping, level: int, types: Set[Type]):
+        if not obj:
+            yield str(obj)
+            return
+
+        next_level = level + 1
+        yield "{\n"
+        for key, value in obj.items():
+            yield spaces * next_level
+            yield from self.write_object(key, next_level, types)
+            yield ": "
+            yield from self.write_object(value, next_level, types)
+            yield ",\n"
+
+        yield f"{spaces * level}}}"
+
+    def write_class(self, obj: Any, level: int, types: Set[Type]):
+        yield f"{obj.__class__.__qualname__}(\n"
+
+        next_level = level + 1
+        for index, f in enumerate(self.context.class_type.get_fields(obj)):
+            if index:
+                yield f",\n{spaces * next_level}{f.name}="
+            else:
+                yield f"{spaces * next_level}{f.name}="
+
+            value = getattr(obj, f.name, types)
+            yield from self.write_object(value, next_level, types)
+
+        yield f"\n{spaces * level})"
diff --git a/xsdata/utils/collections.py b/xsdata/utils/collections.py
index 4ce502a5..a8526265 100644
--- a/xsdata/utils/collections.py
+++ b/xsdata/utils/collections.py
@@ -13,7 +13,7 @@ T = TypeVar("T")
 
 
 def is_array(value: Any) -> bool:
-    if isinstance(value, (list, tuple)):
+    if isinstance(value, (list, tuple, set)):
         return not hasattr(value, "_fields")
 
     return False
diff --git a/xsdata/utils/downloader.py b/xsdata/utils/downloader.py
index b78fdd72..f5b7033f 100644
--- a/xsdata/utils/downloader.py
+++ b/xsdata/utils/downloader.py
@@ -81,7 +81,7 @@ class Downloader:
             self.base_path = Path(uri).parent
             logger.info("Setting base path to %s", self.base_path)
         else:
-            common_path = os.path.commonpath((self.base_path or "", uri))
+            common_path = os.path.commonpath((str(self.base_path) or "", uri))
 
             if common_path:
                 common_path_path = Path(common_path)
diff --git a/xsdata/utils/objects.py b/xsdata/utils/objects.py
index e4676087..eda1ff4a 100644
--- a/xsdata/utils/objects.py
+++ b/xsdata/utils/objects.py
@@ -1,4 +1,7 @@
+import math
 from typing import Any
+from xml.etree.ElementTree import QName
+from xml.sax.saxutils import quoteattr
 
 
 def update(obj: Any, **kwargs: Any):
@@ -15,3 +18,16 @@ def attrsetter(obj: Any, attr: str, value: Any):
         obj = getattr(obj, name)
 
     setattr(obj, last, value)
+
+
+def literal_value(value: Any) -> str:
+    if isinstance(value, str):
+        return quoteattr(value)
+
+    if isinstance(value, float):
+        return str(value) if math.isfinite(value) else f'float("{value}")'
+
+    if isinstance(value, QName):
+        return f'QName("{value.text}")'
+
+    return repr(value).replace("'", '"')


Test patch:
diff --git a/tests/formats/dataclass/serializers/test_code.py b/tests/formats/dataclass/serializers/test_code.py
new file mode 100644
index 00000000..858b6c78
--- /dev/null
+++ b/tests/formats/dataclass/serializers/test_code.py
@@ -0,0 +1,68 @@
+from unittest import TestCase
+
+from tests.fixtures.books.fixtures import books
+from xsdata.formats.dataclass.serializers import PycodeSerializer
+from xsdata.models.enums import Namespace
+
+
+class PycodeSerializerTests(TestCase):
+    def setUp(self) -> None:
+        self.serializer = PycodeSerializer()
+
+    def test_render(self):
+        result = self.serializer.render(books, var_name="books")
+
+        expected = (
+            "from tests.fixtures.books.books import BookForm\n"
+            "from tests.fixtures.books.books import Books\n"
+            "from xsdata.models.datatype import XmlDate\n"
+            "\n"
+            "\n"
+            "books = Books(\n"
+            "    book=[\n"
+            "        BookForm(\n"
+            '            author="Hightower, Kim",\n'
+            '            title="The First Book",\n'
+            '            genre="Fiction",\n'
+            "            price=44.95,\n"
+            "            pub_date=XmlDate(2000, 10, 1),\n"
+            '            review="An amazing story of nothing.",\n'
+            '            id="bk001",\n'
+            '            lang="en"\n'
+            "        ),\n"
+            "        BookForm(\n"
+            '            author="Nagata, Suanne",\n'
+            '            title="Becoming Somebody",\n'
+            '            genre="Biography",\n'
+            "            price=33.95,\n"
+            "            pub_date=XmlDate(2001, 1, 10),\n"
+            '            review="A masterpiece of the fine art of gossiping.",\n'
+            '            id="bk002",\n'
+            '            lang="en"\n'
+            "        ),\n"
+            "    ]\n"
+            ")\n"
+        )
+
+        self.assertEqual(expected, result)
+
+    def test_write_object_with_empty_array(self):
+        iterator = self.serializer.write_object([], 0, set())
+        self.assertEqual("[]", "".join(iterator))
+
+        iterator = self.serializer.write_object((), 0, set())
+        self.assertEqual("()", "".join(iterator))
+
+        iterator = self.serializer.write_object(set(), 0, set())
+        self.assertEqual("set()", "".join(iterator))
+
+    def test_write_object_with_mapping(self):
+        iterator = self.serializer.write_object({}, 0, set())
+        self.assertEqual("{}", "".join(iterator))
+
+        iterator = self.serializer.write_object({"foo": "bar"}, 0, set())
+        self.assertEqual('{\n    "foo": "bar",\n}', "".join(iterator))
+
+    def test_write_object_with_enum(self):
+        iterator = self.serializer.write_object(Namespace.SOAP11, 0, set())
+        self.assertEqual("Namespace.SOAP11", "".join(iterator))