Repository: certbot/certbot

Issue:
UnicodeEncodeError while expanding certificate (ascii)
Whats the problem with the following code?

My domain is: zabanshenas.com, talk.zabanshenas.com

I ran this command: 
```
/usr/local/certbot/certbot-auto -t --renew-by-default --agree-tos --webroot -w /etc/letsencrypt/webroot --server https://acme-v01.api.letsencrypt.org/directory -m eng.hnaseri@gmail.com --expand -d zabanshenas.com,www.zabanshenas.com,talk.zabanshenas.com,www.talk.zabanshenas.com certonly
```


It produced this output: 
```
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for zabanshenas.com
http-01 challenge for www.zabanshenas.com
http-01 challenge for talk.zabanshenas.com
http-01 challenge for www.talk.zabanshenas.com
Using the webroot path /etc/letsencrypt/webroot for all unmatched domains.
Waiting for verification...
Cleaning up challenges
An unexpected error occurred:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 276-280: ordinal not in range(128)
Please see the logfiles in /var/log/letsencrypt for more details.
```
logfile:
```
2017-02-28 16:06:00,465:INFO:certbot.auth_handler:Cleaning up challenges
2017-02-28 16:06:00,465:DEBUG:certbot.plugins.webroot:Removing /etc/letsencrypt/webroot/.well-known/acme-challenge/Olybbkj6iWuPN5MrNF7tAAIGrGyh8Ddy22F0TAoQaRE
2017-02-28 16:06:00,466:DEBUG:certbot.plugins.webroot:Removing /etc/letsencrypt/webroot/.well-known/acme-challenge/5o1e3MiNU8gmtuV_Etz44dwhwQXdFbgR86cZhIHU04c
2017-02-28 16:06:00,467:DEBUG:certbot.plugins.webroot:Removing /etc/letsencrypt/webroot/.well-known/acme-challenge/8kKTJhJa6T5wYMWrEwfr7x1wO2FtcNPMaTTMA27z1uA
2017-02-28 16:06:00,467:DEBUG:certbot.plugins.webroot:All challenges cleaned up, removing /etc/letsencrypt/webroot/.well-known/acme-challenge
2017-02-28 16:06:00,469:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
  File "/root/.local/share/letsencrypt/bin/letsencrypt", line 11, in <module>
    sys.exit(main())
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/main.py", line 882, in main
    return config.func(config, plugins)
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/main.py", line 659, in obtain_cert
    action, _ = _auth_from_available(le_client, config, domains, certname, lineage)
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/main.py", line 104, in _auth_from_available
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/renewal.py", line 296, in renew_cert
    new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/client.py", line 265, in obtain_certificate
    self.config.allow_subset_of_names)
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/auth_handler.py", line 77, in get_authorizations
    self._respond(resp, best_effort)
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/auth_handler.py", line 134, in _respond
    self._poll_challenges(chall_update, best_effort)
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/auth_handler.py", line 197, in _poll_challenges
    _report_failed_challs(all_failed_achalls)
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/auth_handler.py", line 488, in _report_failed_challs
    _generate_failed_chall_msg(achalls), reporter.MEDIUM_PRIORITY)
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/certbot/auth_handler.py", line 504, in _generate_failed_chall_msg
    if messages.is_acme_error(error):
  File "/root/.local/share/letsencrypt/lib/python2.7/site-packages/acme/messages.py", line 39, in is_acme_error
    return (ERROR_PREFIX in str(err)) or (OLD_ERROR_PREFIX in str(err))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 276-280: ordinal not in range(128)
```


Discussion:
huh - looks like the same problem as #4241 
any ideas @bmw?
@zabanshenas, can you include the full log? The error Certbot is struggling to parse should be present in the log before what you've included here.
the full log is here : http://pastebin.com/mYK2uCdV
However I managed to solve the problem.
That was because my subdomain was for a discourse forum and since discourse is installed with docker, I needed to follow [special steps](https://meta.discourse.org/t/how-should-i-enable-letsencrypt-while-discourse-is-beside-other-websites/58107) to make my certificate be issued.
However I think the error I got from certbot wasn't informative enough and in fact it was misleading imo.
Thanks for the log and I'm glad you figured it out. The error you saw was due to our code failing to properly handle the non-ascii characters returned from your server.
So, while this is a misconfiguration with docker, #4241 is a misconfiguration with a service provider. It seems that we're parsing responses without checking them. It might be nice to change the code to validate the response and if it's invalid rather than erroring out due to an ascii parse error, catching it and then giving a more useful error message to the user.
I'm going to flag this as a good volunteer task, but @bmw let me know if you disagree or think that we should make a separate issue for it.
The problem here isn't that the response is invalid, but that it isn't ASCII. We need to make sure we handle non-ascii characters in error messages properly to prevent masking the real error.
Right - I think this is a pretty common problem and we should fix this if we can. Can you add it to a milestone?
Sure. It's in 1.0 now, but I'll add it to 0.14.0 so we can triage it for the next release.
tagging #4516

Resolving patch:
diff --git a/acme/acme/messages.py b/acme/acme/messages.py
--- a/acme/acme/messages.py
+++ b/acme/acme/messages.py
@@ -1,5 +1,6 @@
 """ACME protocol messages."""
 import collections
+import six
 
 from acme import challenges
 from acme import errors
@@ -36,9 +37,13 @@
 
 def is_acme_error(err):
     """Check if argument is an ACME error."""
-    return (ERROR_PREFIX in str(err)) or (OLD_ERROR_PREFIX in str(err))
+    if isinstance(err, Error) and (err.typ is not None):
+        return (ERROR_PREFIX in err.typ) or (OLD_ERROR_PREFIX in err.typ)
+    else:
+        return False
 
 
+@six.python_2_unicode_compatible
 class Error(jose.JSONObjectWithFields, errors.Error):
     """ACME error.
 
@@ -92,10 +97,10 @@ def code(self):
             return code
 
     def __str__(self):
-        return ' :: '.join(
-            part for part in
+        return b' :: '.join(
+            part.encode('ascii', 'backslashreplace') for part in
             (self.typ, self.description, self.detail, self.title)
-            if part is not None)
+            if part is not None).decode()
 
 
 class _Constant(jose.JSONDeSerializable, collections.Hashable):  # type: ignore
diff --git a/certbot/achallenges.py b/certbot/achallenges.py
--- a/certbot/achallenges.py
+++ b/certbot/achallenges.py
@@ -28,7 +28,6 @@
 
 # pylint: disable=too-few-public-methods
 
-
 class AnnotatedChallenge(jose.ImmutableMap):
     """Client annotated challenge.
 


Test patch:
diff --git a/acme/acme/messages_test.py b/acme/acme/messages_test.py
--- a/acme/acme/messages_test.py
+++ b/acme/acme/messages_test.py
@@ -26,6 +26,7 @@ def setUp(self):
             'type': ERROR_PREFIX + 'malformed',
         }
         self.error_custom = Error(typ='custom', detail='bar')
+        self.empty_error = Error()
         self.jobj_custom = {'type': 'custom', 'detail': 'bar'}
 
     def test_default_typ(self):
@@ -45,12 +46,6 @@ def test_description(self):
             'The request message was malformed', self.error.description)
         self.assertTrue(self.error_custom.description is None)
 
-    def test_str(self):
-        self.assertEqual(
-            'urn:ietf:params:acme:error:malformed :: The request message was '
-            'malformed :: foo :: title', str(self.error))
-        self.assertEqual('custom :: bar', str(self.error_custom))
-
     def test_code(self):
         from acme.messages import Error
         self.assertEqual('malformed', self.error.code)
@@ -60,8 +55,16 @@ def test_code(self):
     def test_is_acme_error(self):
         from acme.messages import is_acme_error
         self.assertTrue(is_acme_error(self.error))
-        self.assertTrue(is_acme_error(str(self.error)))
         self.assertFalse(is_acme_error(self.error_custom))
+        self.assertFalse(is_acme_error(self.empty_error))
+        self.assertFalse(is_acme_error("must pet all the {dogs|rabbits}"))
+
+    def test_unicode_error(self):
+        from acme.messages import Error, ERROR_PREFIX, is_acme_error
+        arabic_error = Error(
+                detail=u'\u0639\u062f\u0627\u0644\u0629', typ=ERROR_PREFIX + 'malformed',
+            title='title')
+        self.assertTrue(is_acme_error(arabic_error))
 
     def test_with_code(self):
         from acme.messages import Error, is_acme_error
diff --git a/certbot/tests/errors_test.py b/certbot/tests/errors_test.py
--- a/certbot/tests/errors_test.py
+++ b/certbot/tests/errors_test.py
@@ -23,6 +23,17 @@ def test_str(self):
         self.assertTrue(str(self.error).startswith(
             "Failed authorization procedure. example.com (dns-01): tls"))
 
+    def test_unicode(self):
+        from certbot.errors import FailedChallenges
+        arabic_detail = u'\u0639\u062f\u0627\u0644\u0629'
+        arabic_error = FailedChallenges(set([achallenges.DNS(
+            domain="example.com", challb=messages.ChallengeBody(
+                chall=acme_util.DNS01, uri=None,
+                error=messages.Error(typ="tls", detail=arabic_detail)))]))
+
+        self.assertTrue(str(arabic_error).startswith(
+            "Failed authorization procedure. example.com (dns-01): tls"))
+
 
 class StandaloneBindErrorTest(unittest.TestCase):
     """Tests for certbot.errors.StandaloneBindError."""