Repository: tefra/xsdata

Issue:
UnicodeEncodeError when using WSDL client
I am using xsdata with a wsdl definition to make a request.

I am using this client

`from xsdata.formats.dataclass.client import Client`

and instantiating it like this

`Client.from_service(TalentPortGiveFeedback, location=ENDPOINT)`

However I get the following error when sending unicode characters, such as mandarin, in the body.

`[ERROR] UnicodeEncodeError: 'latin-1' codec can't encode characters in position 609-613: Body ('你好干得好') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.\nTraceback (most recent call last):\n  File 
...
response = client.send(request, headers={\"Authorization\": f\"Bearer {bearer_token}\"})\n  File \"/var/task/xsdata/formats/dataclass/client.py\", line 93, in send\n    response = self.transport.post(self.config.location, data=data, headers=headers)\n  File \"/var/task/xsdata/formats/dataclass/transports.py\", line 44, in post\n    res = requests.post(url, data=data, headers=headers, timeout=self.timeout)\n  File \"/var/task/requests/api.py\", line 115, in post\n    return request(\"post\", url, data=data, json=json, **kwargs)\n  File \"/var/task/requests/api.py\", line 59, in request\n    return session.request(method=method, url=url, **kwargs)\n  File \"/var/task/requests/sessions.py\", line 587, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/var/task/requests/sessions.py\", line 701, in send\n    r = adapter.send(request, **kwargs)\n  File \"/var/task/requests/adapters.py\", line 489, in send\n    resp = conn.urlopen(\n  File \"/var/task/urllib3/connectionpool.py\", line 703, in urlopen\n    httplib_response = self._make_request(\n  File \"/var/task/urllib3/connectionpool.py\", line 398, in _make_request\n    conn.request(method, url, **httplib_request_kw)\n  File \"/var/task/urllib3/connection.py\", line 239, in request\n    super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n  File \"/var/lang/lib/python3.9/http/client.py\", line 1285, in request\n    self._send_request(method, url, body, headers, encode_chunked)\n  File \"/var/lang/lib/python3.9/http/client.py\", line 1330, in _send_request\n    body = _encode(body, 'body')\n  File \"/var/lang/lib/python3.9/http/client.py\", line 168, in _encode\n    raise UnicodeEncodeError("` 

It seems the body is being encoded to 'latin-1' but i cant find any mention of this behavior or how to override it to use 'utf-8' in your docs.

I tried setting headers for charset like this: "Content-Type": "application/xml; charset=utf-8" but to no avail.

Do you have any advice on how to proceed?


Discussion:
sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=tefra_xsdata&pullRequest=787)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=787&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=787&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=787&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=787&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=787&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=787&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=787&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=787&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=787&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=787&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=787&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=787&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=787) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=787&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=787&metric=new_duplicated_lines_density&view=list)


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/tefra/xsdata/pull/787?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Chris+Tsou) Report
Patch coverage: **`100.00`**% and no project coverage change.
> Comparison is base [(`25f9bd4`)](https://app.codecov.io/gh/tefra/xsdata/commit/25f9bd4eea2deec734c21e36ac20015228910956?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Chris+Tsou) 99.96% compared to head [(`ae0ae15`)](https://app.codecov.io/gh/tefra/xsdata/pull/787?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Chris+Tsou) 99.96%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##           master     #787   +/-   ##
=======================================
  Coverage   99.96%   99.96%           
=======================================
  Files         104      104           
  Lines        9107     9111    +4     
  Branches     2036     2037    +1     
=======================================
+ Hits         9104     9108    +4     
  Partials        3        3           
```


| [Impacted Files](https://app.codecov.io/gh/tefra/xsdata/pull/787?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Chris+Tsou) | Coverage Δ | |
|---|---|---|
| [xsdata/formats/dataclass/client.py](https://app.codecov.io/gh/tefra/xsdata/pull/787?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Chris+Tsou#diff-eHNkYXRhL2Zvcm1hdHMvZGF0YWNsYXNzL2NsaWVudC5weQ==) | `100.00% <100.00%> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/tefra/xsdata/pull/787?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Chris+Tsou).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Chris+Tsou).


Patch:
diff --git a/docs/examples/custom-type-mapping.rst b/docs/examples/custom-type-mapping.rst
index 85305fc9..096ee7db 100644
--- a/docs/examples/custom-type-mapping.rst
+++ b/docs/examples/custom-type-mapping.rst
@@ -62,8 +62,9 @@ imported objects, enables xsdata to serialize the models.
 
 
 .. testoutput::
+    :options: +NORMALIZE_WHITESPACE
 
-   <?xml version="1.0" encoding="UTF-8"?>
+    <?xml version="1.0" encoding="UTF-8"?>
     <City>
       <name>footown</name>
       <streets>
diff --git a/docs/json.rst b/docs/json.rst
index 922131ab..014fe853 100644
--- a/docs/json.rst
+++ b/docs/json.rst
@@ -137,6 +137,7 @@ all the imported modules to find a matching dataclass.
 .. doctest::
 
     >>> from tests.fixtures.books import *  # Import all classes
+    >>> from xsdata.formats.dataclass.parsers.config import ParserConfig
     >>> json_string = """{
     ...   "author": "Hightower, Kim",
     ...   "title": "The First Book",
diff --git a/docs/wsdl.rst b/docs/wsdl.rst
index 87ed1132..f9675a93 100644
--- a/docs/wsdl.rst
+++ b/docs/wsdl.rst
@@ -69,8 +69,7 @@ The client can be initialized from the an operation class directly
 
     client = Client.from_service(CalculatorSoapAdd)
     client.config
-    # Config(style='document', location='http://www.dneonline.com/calculator.asmx', transport='http://schemas.xmlsoap.org/soap/http', soap_action='http://tempuri.org/Add', input=<class 'tests.fixtures.calculator.services.CalculatorSoapAddInput'>, output=<class 'tests.fixtures.calculator.services.CalculatorSoapAddOutput'>)
-
+    # Config(style='document', location='http://www.dneonline.com/calculator.asmx', transport='http://schemas.xmlsoap.org/soap/http', soap_action='http://tempuri.org/Add', input=<class 'tests.fixtures.calculator.services.CalculatorSoapAddInput'>, output=<class 'tests.fixtures.calculator.services.CalculatorSoapAddOutput'>, encoding=None)
 
 But you can also override any properties as you see fit
 
@@ -123,3 +122,10 @@ are needed for the webservice to work can not be overwritten.,
 .. code-block::
 
     client.send(params, headers={"User-Agent": "xsdata"})
+
+
+You will need to encode the payload if you intend to send non-ascii characters.
+
+.. code-block::
+
+    client = Client.from_service(encoding="utf-8")
diff --git a/xsdata/formats/dataclass/client.py b/xsdata/formats/dataclass/client.py
index 1d0985d4..8a1252a5 100644
--- a/xsdata/formats/dataclass/client.py
+++ b/xsdata/formats/dataclass/client.py
@@ -32,6 +32,7 @@ class Config(NamedTuple):
     soap_action: str
     input: Type
     output: Type
+    encoding: Optional[str] = None
 
     @classmethod
     def from_service(cls, obj: Any, **kwargs: Any) -> "Config":
@@ -129,4 +130,8 @@ class Client:
                 f"got `{type(obj).__name__}`"
             )
 
-        return self.serializer.render(obj)
+        result = self.serializer.render(obj)
+        if self.config.encoding:
+            return result.encode(self.config.encoding)
+
+        return result


Test patch:
diff --git a/tests/formats/dataclass/test_client.py b/tests/formats/dataclass/test_client.py
index a3c93e1d..045b2fa4 100644
--- a/tests/formats/dataclass/test_client.py
+++ b/tests/formats/dataclass/test_client.py
@@ -5,6 +5,7 @@ from tests.fixtures.calculator import Add
 from tests.fixtures.calculator import CalculatorSoapAdd
 from tests.fixtures.calculator import CalculatorSoapAddInput
 from tests.fixtures.calculator import CalculatorSoapAddOutput
+from tests.fixtures.hello import HelloGetHelloAsString
 from xsdata.exceptions import ClientValueError
 from xsdata.formats.dataclass.client import Client
 from xsdata.formats.dataclass.client import Config
@@ -37,6 +38,7 @@ class ClientTests(TestCase):
             "soap_action": "http://tempuri.org/Add",
             "output": CalculatorSoapAddOutput,
             "transport": "http://schemas.xmlsoap.org/soap/http",
+            "encoding": None,
         }
 
         self.assertEqual(expected, actual)
@@ -88,6 +90,13 @@ class ClientTests(TestCase):
             },
         )
 
+    def test_prepare_payload_with_encoding(self):
+        client = Client.from_service(HelloGetHelloAsString, encoding="utf-8")
+        result = client.prepare_payload(
+            {"Body": {"getHelloAsString": {"arg0": "Χριστόδουλος"}}}
+        )
+        self.assertIn("Χριστόδουλος".encode(), result)
+
     def test_prepare_payload_raises_error_with_type_mismatch(self):
         client = Client.from_service(CalculatorSoapAdd)