Repository: pypa/pipenv

Issue:
prettytoml.tokens.errors.BadEscapeCharacter on Pipfile
##### Issue description

There is a `prettytoml.tokens.errors.BadEscapeCharacter` when I did a `pipenv install -r requirements.txt` on this project https://github.com/cs01/gdbgui.


##### Expected result

Working stuff

##### Actual result

This :
```
Requirements file provided! Importing into Pipfile...
Traceback (most recent call last):
  File "/usr/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==2018.7.1', 'console_scripts', 'pipenv')()
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pipenv/cli.py", line 435, in install
    selective_upgrade=selective_upgrade,
  File "/usr/lib/python3.7/site-packages/pipenv/core.py", line 1890, in do_install
    section = project.editable_packages if not dev else project.editable_dev_packages
  File "/usr/lib/python3.7/site-packages/pipenv/project.py", line 524, in editable_packages
    return self._get_editable_packages(dev=False)
  File "/usr/lib/python3.7/site-packages/pipenv/project.py", line 508, in _get_editable_packages
    for k, v in self.parsed_pipfile.get(section, {}).items()
  File "/usr/lib/python3.7/site-packages/pipenv/project.py", line 509, in <dictcomp>
    if is_editable(v)
  File "/usr/lib/python3.7/site-packages/pipenv/utils.py", line 584, in is_editable
    pipfile_entry.get(key) for key in ('file', 'path') + VCS_LIST
  File "/usr/lib/python3.7/site-packages/pipenv/utils.py", line 584, in <genexpr>
    pipfile_entry.get(key) for key in ('file', 'path') + VCS_LIST
  File "/usr/lib/python3.7/site-packages/pipenv/patched/prettytoml/elements/abstracttable.py", line 73, in get
    return self[key]
  File "/usr/lib/python3.7/site-packages/pipenv/patched/prettytoml/elements/abstracttable.py", line 66, in __getitem__
    for key, value in self.items():
  File "/usr/lib/python3.7/site-packages/pipenv/patched/prettytoml/elements/abstracttable.py", line 34, in items
    yield key.value, value.value
  File "/usr/lib/python3.7/site-packages/pipenv/patched/prettytoml/elements/atomic.py", line 40, in value
    return toml2py.deserialize(self._tokens[self._value_token_index()])
  File "/usr/lib/python3.7/site-packages/pipenv/patched/prettytoml/tokens/toml2py.py", line 33, in deserialize
    return _to_string(token)
  File "/usr/lib/python3.7/site-packages/pipenv/patched/prettytoml/tokens/toml2py.py", line 86, in _to_string
    return _unescape_str(escaped)
  File "/usr/lib/python3.7/site-packages/pipenv/patched/prettytoml/tokens/toml2py.py", line 46, in _unescape_str
    raise BadEscapeCharacter
prettytoml.tokens.errors.BadEscapeCharacter
```

##### Steps to replicate

Do a `pipenv install -r requirements.txt` on a dir with non ascii characters (I think it might be the source)

-------------------------------------------------------------------------------
<details><summary>$ pipenv --support</summary>

Pipenv version: `'2018.7.1'`

Pipenv location: `'/usr/lib/python3.7/site-packages/pipenv'`

Python location: `'/usr/bin/python'`

Other Python installations in `PATH`:

  - `2.7`: `/usr/bin/python2.7`
  - `2.7`: `/usr/bin/python2.7`
  - `3.7`: `/usr/bin/python3.7m`
  - `3.7`: `/usr/bin/python3.7`

  - `3.7.0`: `/usr/bin/python`
  - `2.7.15`: `/usr/bin/python2`
  - `3.7.0`: `/usr/bin/python3`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.17.14-arch1-1-ARCH',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Thu Aug 9 11:56:50 UTC 2018',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'linux'}
```

System environment variables:

  - `LANG`
  - `DISPLAY`
  - `SHLVL`
  - `LOGNAME`
  - `XDG_VTNR`
  - `PWD`
  - `MOZ_PLUGIN_PATH`
  - `ANT_HOME`
  - `XDG_SESSION_ID`
  - `GNOME_TERMINAL_SCREEN`
  - `DESKTOP_SESSION`
  - `XDG_SESSION_DESKTOP`
  - `GDMSESSION`
  - `GNOME_DESKTOP_SESSION_ID`
  - `USERNAME`
  - `WAYLAND_DISPLAY`
  - `DBUS_SESSION_BUS_ADDRESS`
  - `VTE_VERSION`
  - `MAIL`
  - `GNOME_TERMINAL_SERVICE`
  - `GJS_DEBUG_TOPICS`
  - `COLORTERM`
  - `XDG_MENU_PREFIX`
  - `OLDPWD`
  - `GDM_LANG`
  - `XDG_SESSION_TYPE`
  - `GRADLE_HOME`
  - `GJS_DEBUG_OUTPUT`
  - `TERM`
  - `SESSION_MANAGER`
  - `XDG_CURRENT_DESKTOP`
  - `SSH_AUTH_SOCK`
  - `PATH`
  - `QT_IM_MODULE`
  - `HOME`
  - `XDG_SEAT`
  - `XMODIFIERS`
  - `SHELL`
  - `XDG_RUNTIME_DIR`
  - `USER`
  - `_`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_PYTHON_PATH`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/usr/local/bin:/usr/local/sbin:/usr/bin:/opt/cuda/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl`
  - `SHELL`: `/usr/bin/zsh`
  - `LANG`: `fr_FR.UTF-8`
  - `PWD`: `/home/nicoxxl/Projets/Mèche/gdbgui`


---------------------------

Contents of `Pipfile` ('/home/nicoxxl/Projets/Mèche/gdbgui/Pipfile'):

```toml
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
gdbgui = {editable = true, file = "file:///home/nicoxxl/Projets/M\xe8che/gdbgui"}

[dev-packages]

[requires]
python_version = "3.7"

[[source]]
url = "https://pypi.org/simple/"
verify_ssl = true

```

</details>



Discussion:
As you may have noticed in the syntax highlighted block, the `\x` sequence with red background is causing the problem. Try modifying it to the actual path (with the `è` character).
It didn't solve the root cause which is that the Pipfile is automatically generated from my `pipenv install -r requirements.txt`, plus the Pipfile is re-written. 
pipenv as it is doesn't currently support non-unicode paths, we have another issue open on the topic, but that is absolutely the problem here.
@nicoxxl It is unfortunate but [TOML spec](https://github.com/toml-lang/toml#user-content-string) doesn't support `\x` strings. It used to, but has removed the support on later spec version.

The problem is that TOML encoder should convert the string into unicode.

Resolving patch:
diff --git a/pipenv/patched/prettytoml/tokens/py2toml.py b/pipenv/patched/prettytoml/tokens/py2toml.py
--- a/pipenv/patched/prettytoml/tokens/py2toml.py
+++ b/pipenv/patched/prettytoml/tokens/py2toml.py
@@ -2,6 +2,7 @@
 """
 A converter of python values to TOML Token instances.
 """
+from __future__ import unicode_literals
 import codecs
 import datetime
 import six
@@ -81,10 +82,30 @@ def create_string_token(text, bare_string_allowed=False, multiline_strings_allow
 
 
 def _escape_single_line_quoted_string(text):
-    if six.PY2:
-        return text.encode('unicode-escape').encode('string-escape').replace('"', '\\"').replace("\\'", "'")
-    else:
-        return codecs.encode(text, 'unicode-escape').decode().replace('"', '\\"')
+    text = text.decode('utf-8') if isinstance(text, six.binary_type) else text
+    start = 0
+    i = 0
+    res = []
+    _escapes = {'\n': '\\n', '\r': '\\r', '\\': '\\\\', '\t': '\\t',
+                '\b': '\\b', '\f': '\\f', '"': '\\"'}
+
+    def flush():
+        if start < i:
+            res.append(text[start:i])
+        return i + 1
+
+    while i < len(text):
+        c = text[i]
+        if c in _escapes:
+            start = flush()
+            res.append(_escapes[c])
+        elif ord(c) < 0x20:
+            start = flush()
+            res.append('\\u%04x' % ord(c))
+        i += 1
+
+    flush()
+    return ''.join(res)
 
 
 def _create_multiline_string_token(text):
diff --git a/pipenv/patched/prettytoml/tokens/toml2py.py b/pipenv/patched/prettytoml/tokens/toml2py.py
--- a/pipenv/patched/prettytoml/tokens/toml2py.py
+++ b/pipenv/patched/prettytoml/tokens/toml2py.py
@@ -1,3 +1,4 @@
+from __future__ import unicode_literals
 import re
 import string
 import iso8601
@@ -19,7 +20,7 @@ def deserialize(token):
 
     Raises DeserializationError when appropriate.
     """
-    
+
     if token.type == TYPE_BOOLEAN:
         return _to_boolean(token)
     elif token.type == TYPE_INTEGER:
@@ -39,42 +40,40 @@ def _unescape_str(text):
     """
     Unescapes a string according the TOML spec. Raises BadEscapeCharacter when appropriate.
     """
-
-    # Detect bad escape jobs
-    bad_escape_regexp = re.compile(r'([^\\]|^)\\[^btnfr"\\uU]')
-    if bad_escape_regexp.findall(text):
-        raise BadEscapeCharacter
-
-    # Do the unescaping
-    if six.PY2:
-        return _unicode_escaped_string(text).decode('string-escape').decode('unicode-escape')
-    else:
-        return codecs.decode(_unicode_escaped_string(text), 'unicode-escape')
-
-
-def _unicode_escaped_string(text):
-    """
-    Escapes all unicode characters in the given string
-    """
-
-    if six.PY2:
-        text = unicode(text)
-
-    def is_unicode(c):
-        return c.lower() not in string.ascii_letters + string.whitespace + string.punctuation + string.digits
-
-    def escape_unicode_char(x):
-        if six.PY2:
-            return x.encode('unicode-escape')
+    text = text.decode('utf-8') if isinstance(text, six.binary_type) else text
+    tokens = []
+    i = 0
+    basicstr_re = re.compile(r'[^"\\\000-\037]*')
+    unicode_re = re.compile(r'[uU]((?<=u)[a-fA-F0-9]{4}|(?<=U)[a-fA-F0-9]{8})')
+    escapes = {
+        'b': '\b',
+        't': '\t',
+        'n': '\n',
+        'f': '\f',
+        'r': '\r',
+        '\\': '\\',
+        '"': '"',
+        '/': '/',
+        "'": "'"
+    }
+    while True:
+        m = basicstr_re.match(text, i)
+        i = m.end()
+        tokens.append(m.group())
+        if i == len(text) or text[i] != '\\':
+            break
         else:
-            return codecs.encode(x, 'unicode-escape')
-
-    if any(is_unicode(c) for c in text):
-        homogeneous_chars = tuple(escape_unicode_char(c) if is_unicode(c) else c.encode() for c in text)
-        homogeneous_bytes = functools.reduce(operator.add, homogeneous_chars)
-        return homogeneous_bytes.decode()
-    else:
-        return text
+            i += 1
+        if unicode_re.match(text, i):
+            m = unicode_re.match(text, i)
+            i = m.end()
+            tokens.append(six.unichr(int(m.group(1), 16)))
+        else:
+            if text[i] not in escapes:
+                raise BadEscapeCharacter
+            tokens.append(escapes[text[i]])
+            i += 1
+    return ''.join(tokens)
 
 
 def _to_string(token):


Test patch:
diff --git a/tests/unit/test_vendor.py b/tests/unit/test_vendor.py
--- a/tests/unit/test_vendor.py
+++ b/tests/unit/test_vendor.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 # We need to import the patched packages directly from sys.path, so the
 # identity checks can pass.
 import pipenv  # noqa
@@ -8,6 +9,7 @@
 import pytest
 import pytz
 
+import contoml
 from pipfile.api import PipfileParser
 from prettytoml import lexer, tokens
 from prettytoml.elements.atomic import AtomicElement
@@ -104,3 +106,9 @@ def test_inject_environment_variables(self):
 def test_token_date(dt, content):
     token = create_primitive_token(dt)
     assert token == tokens.Token(tokens.TYPE_DATE, content)
+
+
+def test_dump_nonascii_string():
+    content = 'name = "Stažené"\n'
+    toml_content = contoml.dumps(contoml.loads(content))
+    assert toml_content == content