Repository: certbot/certbot

Issue:
none ascii domains, error catching failed
This code should catch UnicodeDecodeError?

```
    # Unicode
    try:
        domain = domain.encode('ascii').lower()
    except UnicodeDecodeError:
        raise errors.ConfigurationError(
            "Internationalized domain names are not presently supported: {0}"
            .format(domain))
```

So why is my log saying:

> 2016-03-13 18:57:09,064:DEBUG:letsencrypt.cli:Exiting abnormally:
> Traceback (most recent call last):
>   File "./letsencrypt", line 11, in <module>
>     sys.exit(main())
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/cli.py", line 1993, in main
>     return config.func(config, plugins)
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/cli.py", line 658, in run
>     domains = _find_domains(config, installer)
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/cli.py", line 133, in _find_domains
>     domains = display_ops.choose_names(installer)
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/display/ops.py", line 208, in choose_names
>     names = get_valid_domains(domains)
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/display/ops.py", line 242, in get_valid_domains
>     valid_domains.append(le_util.enforce_domain_sanity(domain))
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/le_util.py", line 312, in enforce_domain_sanity
>     domain = domain.encode('ascii').lower()
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe5' in position 1: ordinal not in range(128)

row 312 is the `domain = domain.encode('ascii').lower()` row, LE version 0.4.2

none ascii domains, error catching failed
This code should catch UnicodeDecodeError?

```
    # Unicode
    try:
        domain = domain.encode('ascii').lower()
    except UnicodeDecodeError:
        raise errors.ConfigurationError(
            "Internationalized domain names are not presently supported: {0}"
            .format(domain))
```

So why is my log saying:

> 2016-03-13 18:57:09,064:DEBUG:letsencrypt.cli:Exiting abnormally:
> Traceback (most recent call last):
>   File "./letsencrypt", line 11, in <module>
>     sys.exit(main())
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/cli.py", line 1993, in main
>     return config.func(config, plugins)
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/cli.py", line 658, in run
>     domains = _find_domains(config, installer)
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/cli.py", line 133, in _find_domains
>     domains = display_ops.choose_names(installer)
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/display/ops.py", line 208, in choose_names
>     names = get_valid_domains(domains)
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/display/ops.py", line 242, in get_valid_domains
>     valid_domains.append(le_util.enforce_domain_sanity(domain))
>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/le_util.py", line 312, in enforce_domain_sanity
>     domain = domain.encode('ascii').lower()
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe5' in position 1: ordinal not in range(128)

row 312 is the `domain = domain.encode('ascii').lower()` row, LE version 0.4.2



Discussion:
UnicodeDecodeError != UnicodeEncodeError
u"unicode string".encode() --> UnicodeEncodeError
b"byte string".decode() --> UnicodeDecodeError 

oh, missed that one, so guess it should be pathced to "except UnicodeEncodeError:",
tried to replace it with just "except:", but then it just moved the errormessage to  the ".format(domain))" row.

>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/le_util.py", line 318, in enforce_domain_sanity
>     .format(domain))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe5' in position 1: ordinal not in range(128)

changed it to `.format(":".join("{:02x}".format(ord(c)) for c in domain)))` and it seams to work,
not a good solution, so hope someone with python knowledge can patch it.

giving the errors:

> Internationalized domain names are not presently supported: 6d:e5:6c:6b:6f:6e:73:75:6c:74:2e:73:65
> Internationalized domain names are not presently supported: 6d:e5:6c:6b:6f:6e:73:75:6c:74:2e:70:75:67:67:61:6e:2e:73:65

As said the docstring on `enforce_domain_sanity`, `domain` can be either a `unicode` or a `str` so we should both catch `UnicodeDecodeError` and `UnicodeEncodeError`, cf my comment on #2662.

The test result of my pull-request shows some UnicodeDecodeError, the tests maybe should be extended with domains that generate a UnicodeEncodeError too.

@puggan can you describe how to reproduce your original error?

how to reproduce, hmm, can start  with list stuff about my system.
`uname -r -m`: 3.17.7-gentoo x86_64
`python2 --version`: Python 2.7.10
`python --version`: Python 3.4.3
www-servers/apache-2.2.31
LE version 0.4.2, commit 09a649f41749333017fa2363b1ab93d9ed3cc0e2

the vhost with the non ascii chars
[klas.vhos.txt](https://github.com/letsencrypt/letsencrypt/files/172974/klas.vhos.txt)

Running: `/usr/src/letsencrypt/letsencrypt-auto` gave the error

I confirmed this (somewhat) - I think the trouble is that if you get the IDN on the command line, it's a `str`/`bytes` object (decoded Unicode), while if you get the IDN from a vhost, it's a `unicode` object (encoded Unicode). When attempting to detect non-ASCII characters, we can then get _either_ `UnicodeDecodeError` _or_ `UnicodeEncodeError`, depending on what the original type passed to us was.

I don't immediately see the full path to fixing this because when I tried just adding the `UnicodeEncodeError`, I then got an additional error trying to `.format()` the offending domain into the error message string (!). This is a slight subtlety about having a function that can either accept `str` or `unicode` and then try to `.format()` either of them into the same string.

Yes, so it's definitely a matter of whether the name being validated was passed with `-d` (→ `str`, I think) or parsed out of a vhost (→ `unicode`, I think).

Another question is whether, if we did manage to convert this to a `ConfigurationError` in this case as intended, it would apply only to the individual vhost and cause it to be skipped, or if it will currently cause the entire client to give up on any operation if it found a single Unicode vhost anywhere.

UnicodeDecodeError != UnicodeEncodeError
u"unicode string".encode() --> UnicodeEncodeError
b"byte string".decode() --> UnicodeDecodeError 

oh, missed that one, so guess it should be pathced to "except UnicodeEncodeError:",
tried to replace it with just "except:", but then it just moved the errormessage to  the ".format(domain))" row.

>   File "/root/.local/share/letsencrypt/lib64/python2.7/site-packages/letsencrypt/le_util.py", line 318, in enforce_domain_sanity
>     .format(domain))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe5' in position 1: ordinal not in range(128)

changed it to `.format(":".join("{:02x}".format(ord(c)) for c in domain)))` and it seams to work,
not a good solution, so hope someone with python knowledge can patch it.

giving the errors:

> Internationalized domain names are not presently supported: 6d:e5:6c:6b:6f:6e:73:75:6c:74:2e:73:65
> Internationalized domain names are not presently supported: 6d:e5:6c:6b:6f:6e:73:75:6c:74:2e:70:75:67:67:61:6e:2e:73:65

As said the docstring on `enforce_domain_sanity`, `domain` can be either a `unicode` or a `str` so we should both catch `UnicodeDecodeError` and `UnicodeEncodeError`, cf my comment on #2662.

The test result of my pull-request shows some UnicodeDecodeError, the tests maybe should be extended with domains that generate a UnicodeEncodeError too.

@puggan can you describe how to reproduce your original error?

how to reproduce, hmm, can start  with list stuff about my system.
`uname -r -m`: 3.17.7-gentoo x86_64
`python2 --version`: Python 2.7.10
`python --version`: Python 3.4.3
www-servers/apache-2.2.31
LE version 0.4.2, commit 09a649f41749333017fa2363b1ab93d9ed3cc0e2

the vhost with the non ascii chars
[klas.vhos.txt](https://github.com/letsencrypt/letsencrypt/files/172974/klas.vhos.txt)

Running: `/usr/src/letsencrypt/letsencrypt-auto` gave the error

I confirmed this (somewhat) - I think the trouble is that if you get the IDN on the command line, it's a `str`/`bytes` object (decoded Unicode), while if you get the IDN from a vhost, it's a `unicode` object (encoded Unicode). When attempting to detect non-ASCII characters, we can then get _either_ `UnicodeDecodeError` _or_ `UnicodeEncodeError`, depending on what the original type passed to us was.

I don't immediately see the full path to fixing this because when I tried just adding the `UnicodeEncodeError`, I then got an additional error trying to `.format()` the offending domain into the error message string (!). This is a slight subtlety about having a function that can either accept `str` or `unicode` and then try to `.format()` either of them into the same string.

Yes, so it's definitely a matter of whether the name being validated was passed with `-d` (→ `str`, I think) or parsed out of a vhost (→ `unicode`, I think).

Another question is whether, if we did manage to convert this to a `ConfigurationError` in this case as intended, it would apply only to the individual vhost and cause it to be skipped, or if it will currently cause the entire client to give up on any operation if it found a single Unicode vhost anywhere.


Resolving patch:
diff --git a/letsencrypt/le_util.py b/letsencrypt/le_util.py
--- a/letsencrypt/le_util.py
+++ b/letsencrypt/le_util.py
@@ -6,6 +6,7 @@
 import os
 import platform
 import re
+import six
 import socket
 import stat
 import subprocess
@@ -310,10 +311,13 @@ def enforce_domain_sanity(domain):
     # Unicode
     try:
         domain = domain.encode('ascii').lower()
-    except UnicodeDecodeError:
-        raise errors.ConfigurationError(
-            "Internationalized domain names are not presently supported: {0}"
-            .format(domain))
+    except UnicodeError:
+        error_fmt = (u"Internationalized domain names "
+                      "are not presently supported: {0}")
+        if isinstance(domain, six.text_type):
+            raise errors.ConfigurationError(error_fmt.format(domain))
+        else:
+            raise errors.ConfigurationError(str(error_fmt).format(domain))
 
     # Remove trailing dot
     domain = domain[:-1] if domain.endswith('.') else domain


Test patch:
diff --git a/letsencrypt/tests/le_util_test.py b/letsencrypt/tests/le_util_test.py
--- a/letsencrypt/tests/le_util_test.py
+++ b/letsencrypt/tests/le_util_test.py
@@ -323,5 +323,21 @@ def test_help(self):
         self.assertTrue("--old-option" not in stdout.getvalue())
 
 
+class EnforceDomainSanityTest(unittest.TestCase):
+    """Test enforce_domain_sanity."""
+
+    def _call(self, domain):
+        from letsencrypt.le_util import enforce_domain_sanity
+        return enforce_domain_sanity(domain)
+
+    def test_nonascii_str(self):
+        self.assertRaises(errors.ConfigurationError, self._call,
+                          u"eichh\u00f6rnchen.example.com".encode("utf-8"))
+
+    def test_nonascii_unicode(self):
+        self.assertRaises(errors.ConfigurationError, self._call,
+                          u"eichh\u00f6rnchen.example.com")
+
+
 if __name__ == "__main__":
     unittest.main()  # pragma: no cover