Repository: NiklasRosenstein/python-docspec (language: python)

Issue:
`UnicodeEncodeError` when docstring contain non-ascii characters
**Describe the bug**

When a docstrings contains non-ascii character the conversion fails

**To Reproduce**
Steps to reproduce the behavior:
1. create a Python file `foo.py` containing the following:
```py
def foo():
    """
    Some converters tend to recognize clusters of letters as ligatures, such as "ﬀ" (double f)
    """
    pass
```

2. from the same folder, run `pydoc-markdown -I . -m foo`
3. see the error:
```
Traceback (most recent call last):
  File "/Users/massi/.virtualenvs/haystack/bin/pydoc-markdown", line 8, in <module>
    sys.exit(cli())
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/pydoc_markdown/main.py", line 383, in cli
    session.render(pydocmd)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/pydoc_markdown/main.py", line 132, in render
    modules = config.load_modules()
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/pydoc_markdown/__init__.py", line 150, in load_modules
    modules.extend(loader.load())
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/docspec_python/__init__.py", line 90, in load_python_modules
    yield parse_python_module(filename, module_name=module_name, options=options, encoding=encoding)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/docspec_python/__init__.py", line 131, in parse_python_module
    return parse_python_module(fpobj, fp, module_name, options, encoding)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/docspec_python/__init__.py", line 136, in parse_python_module
    return parser.parse(ast, filename, module_name)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/docspec_python/parser.py", line 300, in parse
    member = self.parse_declaration(module, node)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/docspec_python/parser.py", line 326, in parse_declaration
    return self.parse_funcdef(parent, node, False, decorations)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/docspec_python/parser.py", line 529, in parse_funcdef
    docstring = self.get_docstring_from_first_node(body)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/docspec_python/parser.py", line 799, in get_docstring_from_first_node
    return self.prepare_docstring(get_value(node.children[0]), parent)
  File "/Users/massi/.virtualenvs/haystack/lib/python3.10/site-packages/docspec_python/parser.py", line 898, in prepare_docstring
    docstring.content = docstring.content.encode("latin1").decode(
UnicodeEncodeError: 'latin-1' codec can't encode character '\ufb00' in position 77: ordinal not in range(256)
```

**Expected behavior**
No errors like it was with version<2.1.0




Discussion / hints:
Hey @masci, thanks for the bug report. Dang, it seems I didn't test this sufficiently and trusted StackOverflow a bit too much 👀 

The encode/decode code here was introduced to convert a Python literal string into an actual string as it would be parsed by the Python interpreter to memory (so when you write `"foo\n"` into your docstring, would actually be `"foo\n"` in the `Docstring.content` instead of `"foo\\n"`)

Unless there's another better working solution using the encode/decode logic, I suppose we need to manually parse the string and convert special character sequences.
Thanks for following up! I'm not sure I get 100% the logic of the answer in SO but at some point I see
```py
...
s.encode('latin1')         # To bytes, required by 'unicode-escape'
...
```
and I wonder, if the goal of that step is just to have bytes out of the original string, can't we just encode using something more flexible than `latin1`, like `utf-8`? Am I missing something?
The reason is that `latin1` and `unicode_escape` seem to have a convenient overlap in escape character use, or something like that. But if `latin1` can't encode everything, then it's no use either. 🤦 

```
>>> 'ü'.encode('latin1')
b'\xfc'
>>> 'ü'.encode('latin1').decode('unicode_escape')
'ü'
>>> 'ü'.encode('utf-8')
b'\xc3\xbc'
>>> 'ü'.encode('utf-8').decode('unicode_escape')
'Ã¼'
```
It seems like you already found the PR and thus the StackOverflow answer I was referring to, but for reference: #83 and https://stackoverflow.com/a/58829514/791713

The best alternative that I can think of without re-implementing the decoding of raw strings is to use `ast.literal_eval()`. Actually that does appear rather elegant to me, in particular because the string we're dealing with will have the quotes around it.

```
    if s:
      s = ast.literal_eval(s)
      return Docstring(location, dedent_docstring(s).strip())
```


Fix patch:
diff --git a/docspec-python/.changelog/_unreleased.toml b/docspec-python/.changelog/_unreleased.toml
new file mode 100644
index 0000000..5a6c76e
--- /dev/null
+++ b/docspec-python/.changelog/_unreleased.toml
@@ -0,0 +1,8 @@
+[[entries]]
+id = "a519ecc4-1b48-4883-a930-dc6f7196e54d"
+type = "fix"
+description = "Fix #91 by using `ast.literal_eval()` instead of `encode(latin1).decode(unicode_escape)` method."
+author = "@NiklasRosenstein"
+issues = [
+    "https://github.com/NiklasRosenstein/docspec/issues/91",
+]
diff --git a/docspec-python/src/docspec_python/parser.py b/docspec-python/src/docspec_python/parser.py
index 63a9eb0..2fc71a3 100644
--- a/docspec-python/src/docspec_python/parser.py
+++ b/docspec-python/src/docspec_python/parser.py
@@ -27,6 +27,7 @@
 
 import dataclasses
 from io import StringIO
+import ast
 import os
 import re
 import sys
@@ -822,19 +823,10 @@ def prepare_docstring(self, s: str, node_for_location: NL) -> t.Optional[Docstri
         new_line = ' ' * max(0, len(line) - len(new_line) - initial_indent) + new_line
         lines.append(new_line.rstrip())
       return Docstring(location, '\n'.join(lines).strip())
-    is_raw = False
-    while s and s[0] in ('rfub'):
-      if s[0] == 'r':
-        is_raw = True
-        s = s[1:]
-    docstring: t.Optional[Docstring] = None
-    if s.startswith('"""') or s.startswith("'''"):
-      docstring = Docstring(location, dedent_docstring(s[3:-3]).strip())
-    elif s.startswith('"') or s.startswith("'"):
-      docstring = Docstring(location, dedent_docstring(s[1:-1]).strip())
-    if docstring and not is_raw:
-      docstring.content = docstring.content.encode('latin1').decode('unicode_escape', errors='ignore')
-    return docstring
+    if s:
+      s = ast.literal_eval(s)
+      return Docstring(location, dedent_docstring(s).strip())
+    return None
 
   def nodes_to_string(self, nodes: list[NL]) -> str:
     """


Test patch:
diff --git a/docspec-python/test/test_parser.py b/docspec-python/test/test_parser.py
index b64084e..93960b7 100644
--- a/docspec-python/test/test_parser.py
+++ b/docspec-python/test/test_parser.py
@@ -57,6 +57,7 @@ def unset_location(obj: HasLocation):
     for arg in obj.args:
       unset_location(arg)
 
+
 def docspec_test(module_name=None, parser_options=None, strip_locations: bool = True):
   """
   Decorator for docspec unit tests.
@@ -478,6 +479,7 @@ def build_docker_image(
       ),
     ]
 
+
 @docspec_test()
 def test_funcdef_with_match_statement():
     """
@@ -516,12 +518,16 @@ def single():
 
   def multi():
     r''' M\\lti raw docstring. '''
+
+  def special_characters():
+    ''' ﬀ '''
   """
 
   return [
     mkfunc("normal", "Normal d\\cstring.", 0, []),
     mkfunc("single", "S\\\\ngle raw docstring.", 0, []),
-    mkfunc("multi", "M\\\\lti raw docstring.", 0, [])
+    mkfunc("multi", "M\\\\lti raw docstring.", 0, []),
+    mkfunc("special_characters", "ﬀ", 0, [])
   ]