Repository: pyca/cryptography

Issue:
Add RFC4514 parser
This PR adds a RFC4514 parser.

The implementation uses [shlex.shlex](https://docs.python.org/3/library/shlex.html) parsers to take care of quoting and escaping issues. I've added quite a few tests and I think the approach is maybe not perfect, but reasonably safe.

Known limitations:

1. The parser does not understand unicode characters (like in the example given in RFC 4514, section 4 in the last example). However, neither does the serialization already implemented, so I think it's a fair limitation that may be fixed in a separate pull request.
2. Using `shlex` makes it possible to quote strings, so `CN="Foo,Bar"` will work as a user would probably expect (with the CN being `Foo,Bar` without quotes), but the string is not strictly RFC4514 compatible anyway. The standard would mandate `"` to be escaped, but this would also change the semantics, as it would make them part of the CN.

Fixes #6165.
[Feature request] Parse DN string into list of NameAttribute's
There seems to be no way to convert a string (or bytestring) containing a Distinguished Name or X.509 Subject into a list of corresponding NameAttributes, suitable for (example) passing to `x509.CertificateSigningRequestBuilder().subject_name()`.

There's the `rfc4154_string()` which goes in the opposite direction, however.

Using Python 3.8.2 on MacOS 11.3.1, `cryptography` 3.4.7 (installed via `pip3`).


Discussion:
Hi @alex, 

Thanks for the review so far. I solved the two minor issues you mentioned (string formatting, compiled regex).

I don't know how "nervous" you're about using `shlex`. I could implement a manual parsing solution, but I'm sure it would be significantly more code.

kr, Mat
First, I'm sorry we've sat on this for so long without a forward direction :-(

After long review, I've a) gotten comfortable with the idea of us including this API (despite how much I hate the format :-)), b) I'm not comfortable with `shlex` and would prefer a dedicated parser for this format that's easier to ensure matches the RFC.

If you're still up for working on this, awesome, otherwise I can port this away from `shlex`. I'm hoping we can include this in our next release.
Fyi,
RFC4514 is an LDAP thing.
So, maybe some idea's can be gathered from here: 

- https://github.com/python-ldap/python-ldap/blob/master/Lib/ldap/dn.py
- https://github.com/cannatag/ldap3/blob/dev/ldap3/utils/dn.py
Here's most of a naïve implementation:

```
from cryptography import x509
from cryptography.x509.oid import NameOID

import re

oidMap = {
    'CN': NameOID.COMMON_NAME,
    'C': NameOID.COUNTRY_NAME,
    'L': NameOID.LOCALITY_NAME,
    'ST': NameOID.STATE_OR_PROVINCE_NAME,
    'SP': NameOID.STATE_OR_PROVINCE_NAME,
    'S': NameOID.STATE_OR_PROVINCE_NAME,
    'STREET': NameOID.STREET_ADDRESS,
    'O': NameOID.ORGANIZATION_NAME,
    'OU': NameOID.ORGANIZATIONAL_UNIT_NAME,
    'SN': NameOID.SURNAME,
    'GN': NameOID.GIVEN_NAME,
    'T': NameOID.TITLE,
    'MAIL': NameOID.EMAIL_ADDRESS,
    'E': NameOID.EMAIL_ADDRESS,
    'PC': NameOID.POSTAL_CODE,
}

def parse_dn(s):
    """Parse a string into a set of OIDs"""

    attrs = []

    if s[0:1] != '/':
       raise ValueError
    s = s[1:]

    # need to handle escaping of the RHS, etc.
    prog = re.compile(u'([A-Z]+)=(.*)')
    for pair in s.split('/'):
        match = prog.fullmatch(pair)
        if match:
           (name, value) = match.group(1, 2)
           if not name in oidMap:
               raise ValueError
           attrs.append(x509.NameAttribute(oidMap[name], value))

    return attrs
```
Duplicate of #6165?

Okay, redone leveraging `str2dn` from `ldap.dn`:

```
from cryptography import x509
from cryptography.x509.oid import NameOID

import ldap.dn

import typing

# _NAME_TO_NAMEOID = { i.value: i for i in _NAMEOID_TO_NAME }

_NAME_TO_NAMEOID = {
    'CN': NameOID.COMMON_NAME,
    'L': NameOID.LOCALITY_NAME,
    'ST': NameOID.STATE_OR_PROVINCE_NAME,
    'O': NameOID.ORGANIZATION_NAME,
    'OU': NameOID.ORGANIZATIONAL_UNIT_NAME,
    'C': NameOID.COUNTRY_NAME,
    'STREET': NameOID.STREET_ADDRESS,
    'DC': NameOID.DOMAIN_COMPONENT,
    'UID': NameOID.USER_ID,
}

def parse_dn(s: bytes) -> x509.Name:
    """Parse a string into a set of OID/value pairs"""

    attrs = []

    for tup in ldap.dn.str2dn(s, ldap.DN_FORMAT_LDAPV3):
        (name, value, flags) = tup[0]

        if not name in _NAME_TO_NAMEOID:
            raise ValueError('Unknown OID in Distinguished Name', name)

        attrs.append(x509.NameAttribute(_NAME_TO_NAMEOID[name], value))

    return x509.Name(attrs)

```
I don't like the idea of having code that tries to map stringly typed things like this into more structured formats. Yes there exists a spec, but I strongly suspect that there exist dozens of slightly incompatible variants, each of which people want to work. I think I'd rather this was a mapping that users built themselves so that they can ensure it is fully compatible with whatever quirky implementation they're actually relying on. (For the record, I regret landing `rfc4154_string`)
>  I think I'd rather this was a mapping that users built themselves so that they can ensure it is fully compatible with whatever quirky implementation they're actually relying on.

_One ring to rule them all._

That's the nice thing about reference implementations: it gives people a **correct** way of doing something conveniently in their reach so they're not tempted to stray down the path of brokenness and divergence, at least not without a lot of additional unwarranted effort.

Well, instead of "*in the darkness bind them*", it's light to keep them on the righteous path.
> code that tries to map stringly typed things like this into more structured formats.

I think you're describing all serialization formats. :) I view RFC 4514 as a just another serialization format in the cryptography domain.

You can have non-conforming implementations and vendor extensions with binary formats like X.509 as well. Like negative serial numbers (#4843), negative RSA moduli (https://bugs.chromium.org/p/chromium/issues/detail?id=532048), I'm sure there are dozens of examples. And sometimes those situations require difficult decisions. That's just the reality of interoperability.

If `cryptography` library doesn't accept RFC 4514 support in core, users who need it will probably implement it themselves, and most likely do a worse job than this project could. I think that's a net loss for everyone -- duplicated effort and worse interoperability.

FTR, I proposed a solution in #6166. I'm open to providing a more traditional parser solution that does not use `shlex` as well (I just think it's essentially going to be the same result).

Resolving patch:
diff --git a/src/cryptography/x509/name.py b/src/cryptography/x509/name.py
--- a/src/cryptography/x509/name.py
+++ b/src/cryptography/x509/name.py
@@ -3,6 +3,8 @@
 # for complete details.
 
 import binascii
+import re
+import sys
 import typing
 import warnings
 
@@ -53,6 +55,7 @@ class _ASN1Type(utils.Enum):
     NameOID.DOMAIN_COMPONENT: "DC",
     NameOID.USER_ID: "UID",
 }
+_NAME_TO_NAMEOID = {v: k for k, v in _NAMEOID_TO_NAME.items()}
 
 
 def _escape_dn_value(val: typing.Union[str, bytes]) -> str:
@@ -84,6 +87,33 @@ def _escape_dn_value(val: typing.Union[str, bytes]) -> str:
     return val
 
 
+_HEX_ESCAPE_RE = re.compile(r"\\([\da-zA-Z]{2})")
+
+
+def _unescape_dn_value(val: str) -> str:
+    if not val:
+        return ""
+
+    # See https://tools.ietf.org/html/rfc4514#section-3
+
+    # special = escaped / SPACE / SHARP / EQUALS
+    # escaped = DQUOTE / PLUS / COMMA / SEMI / LANGLE / RANGLE
+    val = (
+        val.replace("\\\\", "\\")
+        .replace('\\"', '"')
+        .replace("\\+", "+")
+        .replace("\\,", ",")
+        .replace("\\;", ";")
+        .replace("\\<", "<")
+        .replace("\\>", ">")
+        .replace("\\ ", " ")
+        .replace("\\#", "#")
+        .replace("\\=", "=")
+    )
+
+    return _HEX_ESCAPE_RE.sub(lambda m: chr(int(m.group(1), 16)), val)
+
+
 class NameAttribute:
     def __init__(
         self,
@@ -273,6 +303,10 @@ def __init__(
                 " or a list RelativeDistinguishedName"
             )
 
+    @classmethod
+    def from_rfc4514_string(cls, data: str) -> "Name":
+        return _RFC4514NameParser(data).parse()
+
     def rfc4514_string(
         self, attr_name_overrides: typing.Optional[_OidNameMap] = None
     ) -> str:
@@ -325,3 +359,92 @@ def __len__(self) -> int:
     def __repr__(self) -> str:
         rdns = ",".join(attr.rfc4514_string() for attr in self._attributes)
         return "<Name({})>".format(rdns)
+
+
+class _RFC4514NameParser:
+    _OID_RE = re.compile(r"(\d|[1-9]\d+)(\.\d|[1-9]\d+)+")
+    _DESCR_RE = re.compile(r"[a-zA-Z][a-zA-Z\d-]*")
+
+    _PAIR = r"\\([\\ #=\"\+,;<>]|[\da-zA-Z]{2})"
+    _LUTF1 = r"[\x01-\x1f\x21\x24-\x2A\x2D-\x3A\x3D\x3F-\x5B\x5D-\x7F]"
+    _SUTF1 = r"[\x01-\x21\x23-\x2A\x2D-\x3A\x3D\x3F-\x5B\x5D-\x7F]"
+    _TUTF1 = r"[\x01-\x1F\x21\x23-\x2A\x2D-\x3A\x3D\x3F-\x5B\x5D-\x7F]"
+    _UTFMB = rf"[\x80-{chr(sys.maxunicode)}]"
+    _LEADCHAR = rf"{_LUTF1}|{_UTFMB}"
+    _STRINGCHAR = rf"{_SUTF1}|{_UTFMB}"
+    _TRAILCHAR = rf"{_TUTF1}|{_UTFMB}"
+    _STRING_RE = re.compile(
+        rf"""
+        (
+            ({_LEADCHAR}|{_PAIR})
+            (
+                ({_STRINGCHAR}|{_PAIR})*
+                ({_TRAILCHAR}|{_PAIR})
+            )?
+        )?
+        """,
+        re.VERBOSE,
+    )
+    _HEXSTRING_RE = re.compile(r"#([\da-zA-Z]{2})+")
+
+    def __init__(self, data: str) -> None:
+        self._data = data
+        self._idx = 0
+
+    def _has_data(self) -> bool:
+        return self._idx < len(self._data)
+
+    def _peek(self) -> typing.Optional[str]:
+        if self._has_data():
+            return self._data[self._idx]
+        return None
+
+    def _read_char(self, ch: str) -> None:
+        if self._peek() != ch:
+            raise ValueError
+        self._idx += 1
+
+    def _read_re(self, pat) -> str:
+        match = pat.match(self._data, pos=self._idx)
+        if match is None:
+            raise ValueError
+        val = match.group()
+        self._idx += len(val)
+        return val
+
+    def parse(self) -> Name:
+        rdns = [self._parse_rdn()]
+
+        while self._has_data():
+            self._read_char(",")
+            rdns.append(self._parse_rdn())
+
+        return Name(rdns)
+
+    def _parse_rdn(self) -> RelativeDistinguishedName:
+        nas = [self._parse_na()]
+        while self._peek() == "+":
+            self._read_char("+")
+            nas.append(self._parse_na())
+
+        return RelativeDistinguishedName(nas)
+
+    def _parse_na(self) -> NameAttribute:
+        try:
+            oid_value = self._read_re(self._OID_RE)
+        except ValueError:
+            name = self._read_re(self._DESCR_RE)
+            oid = _NAME_TO_NAMEOID.get(name)
+            if oid is None:
+                raise ValueError
+        else:
+            oid = ObjectIdentifier(oid_value)
+
+        self._read_char("=")
+        if self._peek() == "#":
+            value = self._read_re(self._HEXSTRING_RE)
+            value = binascii.unhexlify(value[1:]).decode()
+        else:
+            value = _unescape_dn_value(self._read_re(self._STRING_RE))
+
+        return NameAttribute(oid, value)


Test patch:
diff --git a/tests/hypothesis/test_x509.py b/tests/hypothesis/test_x509.py
new file mode 100644
--- /dev/null
+++ b/tests/hypothesis/test_x509.py
@@ -0,0 +1,22 @@
+# This file is dual licensed under the terms of the Apache License, Version
+# 2.0, and the BSD License. See the LICENSE file in the root of this repository
+# for complete details.
+
+from hypothesis import HealthCheck, example, given, settings
+from hypothesis.strategies import text
+
+from cryptography import x509
+
+
+@settings(suppress_health_check=[HealthCheck.too_slow], deadline=None)
+@given(text())
+@example("CN=cryptography.io")
+def test_name_from_rfc4514(data):
+    try:
+        x509.Name.from_rfc4514_string(data)
+    except ValueError:
+        return
+
+    # Can't assert that it round-trips because of things like "OID=value"
+    # where OID is one of the known OIDs that serializes to a known value
+    # (e.g. CN)
diff --git a/tests/x509/test_name.py b/tests/x509/test_name.py
new file mode 100644
--- /dev/null
+++ b/tests/x509/test_name.py
@@ -0,0 +1,153 @@
+# -*- coding: utf-8 -*-
+# This file is dual licensed under the terms of the Apache License, Version
+# 2.0, and the BSD License. See the LICENSE file in the root of this repository
+# for complete details.
+
+
+import pytest
+
+from cryptography.x509 import (
+    Name,
+    NameAttribute,
+    NameOID,
+    RelativeDistinguishedName,
+)
+
+
+class TestRFC4514:
+    def test_invalid(self, subtests):
+        for value in [
+            "C=US,CN=Joe , Smith,DC=example",
+            ",C=US,CN=Joe , Smith,DC=example",
+            "C=US,UNKNOWN=Joe , Smith,DC=example",
+            "C=US,CN,DC=example",
+            "C=US,FOOBAR=example",
+        ]:
+            with subtests.test():
+                with pytest.raises(ValueError):
+                    Name.from_rfc4514_string(value)
+
+    def test_valid(self, subtests):
+        for value, expected in [
+            (
+                r"CN=James \"Jim\" Smith\, III",
+                Name(
+                    [
+                        NameAttribute(
+                            NameOID.COMMON_NAME, 'James "Jim" Smith, III'
+                        )
+                    ]
+                ),
+            ),
+            (
+                r"UID=\# escape\+\,\;\00this\ ",
+                Name([NameAttribute(NameOID.USER_ID, "# escape+,;\0this ")]),
+            ),
+            (
+                r"2.5.4.3=James \"Jim\" Smith\, III",
+                Name(
+                    [
+                        NameAttribute(
+                            NameOID.COMMON_NAME, 'James "Jim" Smith, III'
+                        )
+                    ]
+                ),
+            ),
+            ("ST=", Name([NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "")])),
+            (
+                "OU=Sales+CN=J.  Smith,DC=example,DC=net",
+                Name(
+                    [
+                        RelativeDistinguishedName(
+                            [
+                                NameAttribute(
+                                    NameOID.ORGANIZATIONAL_UNIT_NAME, "Sales"
+                                ),
+                                NameAttribute(
+                                    NameOID.COMMON_NAME, "J.  Smith"
+                                ),
+                            ]
+                        ),
+                        RelativeDistinguishedName(
+                            [
+                                NameAttribute(
+                                    NameOID.DOMAIN_COMPONENT, "example"
+                                )
+                            ]
+                        ),
+                        RelativeDistinguishedName(
+                            [NameAttribute(NameOID.DOMAIN_COMPONENT, "net")]
+                        ),
+                    ]
+                ),
+            ),
+            (
+                "CN=cryptography.io,O=PyCA,L=,ST=,C=US",
+                Name(
+                    [
+                        NameAttribute(NameOID.COMMON_NAME, "cryptography.io"),
+                        NameAttribute(NameOID.ORGANIZATION_NAME, "PyCA"),
+                        NameAttribute(NameOID.LOCALITY_NAME, ""),
+                        NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, ""),
+                        NameAttribute(NameOID.COUNTRY_NAME, "US"),
+                    ]
+                ),
+            ),
+            (
+                r"C=US,CN=Joe \, Smith,DC=example",
+                Name(
+                    [
+                        NameAttribute(NameOID.COUNTRY_NAME, "US"),
+                        NameAttribute(NameOID.COMMON_NAME, "Joe , Smith"),
+                        NameAttribute(NameOID.DOMAIN_COMPONENT, "example"),
+                    ]
+                ),
+            ),
+            (
+                r"C=US,CN=Jane \"J\,S\" Smith,DC=example",
+                Name(
+                    [
+                        NameAttribute(NameOID.COUNTRY_NAME, "US"),
+                        NameAttribute(NameOID.COMMON_NAME, 'Jane "J,S" Smith'),
+                        NameAttribute(NameOID.DOMAIN_COMPONENT, "example"),
+                    ]
+                ),
+            ),
+            (
+                'C=US,CN=\\"Jane J\\,S Smith\\",DC=example',
+                Name(
+                    [
+                        NameAttribute(NameOID.COUNTRY_NAME, "US"),
+                        NameAttribute(NameOID.COMMON_NAME, '"Jane J,S Smith"'),
+                        NameAttribute(NameOID.DOMAIN_COMPONENT, "example"),
+                    ]
+                ),
+            ),
+            (
+                'C=US,CN=\\"Jane \\"J\\,S\\" Smith\\",DC=example',
+                Name(
+                    [
+                        NameAttribute(NameOID.COUNTRY_NAME, "US"),
+                        NameAttribute(
+                            NameOID.COMMON_NAME, '"Jane "J,S" Smith"'
+                        ),
+                        NameAttribute(NameOID.DOMAIN_COMPONENT, "example"),
+                    ]
+                ),
+            ),
+            (
+                r"C=US,CN=Jane=Smith,DC=example",
+                Name(
+                    [
+                        NameAttribute(NameOID.COUNTRY_NAME, "US"),
+                        NameAttribute(NameOID.COMMON_NAME, "Jane=Smith"),
+                        NameAttribute(NameOID.DOMAIN_COMPONENT, "example"),
+                    ]
+                ),
+            ),
+            (r"CN=#616263", Name([NameAttribute(NameOID.COMMON_NAME, "abc")])),
+            (r"CN=👍", Name([NameAttribute(NameOID.COMMON_NAME, "👍")])),
+        ]:
+            with subtests.test():
+                result = Name.from_rfc4514_string(value)
+                assert result == expected