Repository: certbot/certbot

Issue:
UnicodeEncodeError in nginxparser.py from RawNginxDumper
If you're having trouble using Certbot and aren't sure you've found a bug or
request for a new feature, please first try asking for help at
https://community.letsencrypt.org/. There is a much larger community there of
people familiar with the project who will be able to more quickly answer your
questions.

## My operating system is (include version):

CentOS 7, Digital Ocean droplet

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):

certbot-auto

## I ran this command and it produced this output:

certbot-auto

gave correct domain and everything, certificate works, will not install.  Log says there's a unicode error in RawNginxDumper

## Certbot's behavior differed from what I expected because:

It should just work since nginx seems to have no issues...

## Here is a Certbot log showing the issue (if available):
###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

```
2020-07-16 20:50:55,377:INFO:certbot_nginx._internal.configurator:Deploying Certificate to VirtualHost /etc/nginx/conf.d/default.conf
2020-07-16 20:50:55,377:DEBUG:certbot.reverter:Creating backup of /etc/nginx/nginx.conf
2020-07-16 20:50:55,378:DEBUG:certbot.reverter:Creating backup of /etc/nginx/conf.d/default.conf
2020-07-16 20:50:55,378:DEBUG:certbot.reverter:Creating backup of /etc/nginx/mime.types
2020-07-16 20:50:55,378:DEBUG:certbot.reverter:Creating backup of /etc/letsencrypt/options-ssl-nginx.conf
2020-07-16 20:50:55,380:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/_internal/client.py", line 521, in deploy_certificate
    self.installer.save()  # needed by the Apache plugin
  File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot_nginx/_internal/configurator.py", line 1051, in save
    self.parser.filedump(ext='')
  File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot_nginx/_internal/parser.py", line 250, in filedump
    out = nginxparser.dumps(tree)
  File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot_nginx/_internal/nginxparser.py", line 140, in dumps
    return str(RawNginxDumper(blocks.spaced))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 662-663: ordinal not in range(128)
```

## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:

nginx conf:

server {
    listen 80;
    server_name my-domain.org;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/user/project/phencards.sock;
    }
}

Certbot also added all the default stuff like `listen 443 ssl` the cert locations etc.





Discussion:
Could you please dump the raw nginx config which produces this error, and attach it here?

    sudo nginx -T > /tmp/full-nginx.conf

This is/was a Certbot bug where the nginx parser was less tolerant of Unicode than it should be, but I thought it had been fixed since v1.4.0 (#7797). 
```
# configuration file /etc/nginx/nginx.conf:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/default.conf:
#server {
#    listen       80;
#    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;

#    location / {
#        root   /usr/share/nginx/html;
#        index  index.html index.htm;
#    }

server {
    server_name phencards-dev.wglab.org;
    access_log  /var/log/nginx/host.access.log  main;
    set $upstream 157.245.2.226:5005;

    location / {
	proxy_pass_header Authorization;
        proxy_pass  http://$upstream;
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_http_version 1.1;
	proxy_set_header Connection “”;
	proxy_buffering off;
	client_max_body_size 0;
	proxy_read_timeout 36000s;
	proxy_redirect off;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/phencards-dev.wglab.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/phencards-dev.wglab.org/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


}



server {
    if ($host = phencards-dev.wglab.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name phencards-dev.wglab.org;
    return 404; # managed by Certbot


}

# configuration file /etc/letsencrypt/options-ssl-nginx.conf:
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file.

ssl_session_cache shared:le_nginx_SSL:10m;
ssl_session_timeout 1440m;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;

ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";


```
Well, the Unicode in that file which is likely the cause of the error is:

>	proxy_set_header Connection “”;

Those quotes are U+201C and U+201D, rather than the usual ASCII ones. Possibly you copied this from the Let's Encrypt community forum, where the forum software converts them unlesss they're in code blocks.

Nonetheless, Certbot should not crash on this, because exactly this behavior has already been fixed in the past. I can't reproduce it on latest `certbot-auto`. Your `certbot-auto` is not being held at a previous version, is it?

    sudo certbot-auto --version
Yep that was it, thanks.  I feel dumb.  Nice work.

version 1.6.0 btw.



On Thu, Jul 16, 2020 at 7:28 PM alexzorin <notifications@github.com> wrote:

> Well, the Unicode in that file which is likely the cause of the error is:
>
> proxy_set_header Connection “”;
>
> Those quotes are U+201C and U+201D, rather than the usual ASCII ones.
> Possibly you copied this from the Let's Encrypt community forum, where the
> forum software converts them unlesss they're in code blocks.
>
> Nonetheless, Certbot should not crash on this, because exactly this
> behavior has already been fixed in the past. I can't reproduce it on latest
> certbot-auto. Your certbot-auto is not being held at a previous version,
> is it?
>
> sudo certbot-auto --version
>
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/certbot/certbot/issues/8146#issuecomment-659732211>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABSDYBEBA3QROQIHFPO6PKTR36EIHANCNFSM4O45XYPQ>
> .
>

Actually, I managed to reproduce this on CentOS 7 + `certbot-auto` 1.6.0. 

It might be a Python 2 vs Python 3 problem.
You were right.  I changed the quotes and it no longer threw an error.

On Thu, Jul 16, 2020 at 7:40 PM alexzorin <notifications@github.com> wrote:

> Actually, I managed to reproduce this on CentOS 7 + certbot-auto 1.6.0.
>
> It might be a Python 2 vs Python 3 problem.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/certbot/certbot/issues/8146#issuecomment-659736615>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABSDYBCYQTN6KW4QLL3LSETR36FYJANCNFSM4O45XYPQ>
> .
>

I have a similar error,

    UnicodeEncodeError: 'ascii' codec can't encode characters in position 436-451: ordinal not in range(128)

when running 

    certbot --nginx

on CentOS 7. Surprisingly, that is a recent package (1.9.0). 
It prohibits any non-ascii data in nginx configuration file for my site. However, my site uses unicode folders for media files, so I can't simply change the domain name to punicode. I delete unicode, run certbot, and then restore the proper paths.

Apart from that, everything is great, thanks a lot for the program!
However, it turns out that I can't automatically renew this certificate any longer.

    certbot renew --dry-run

(from the log file):

    File "/usr/lib/python2.7/site-packages/certbot_nginx/_internal/nginxparser.py", line 140, in dumps
        return str(RawNginxDumper(blocks.spaced))
    UnicodeEncodeError: 'ascii' codec can't encode characters in position 587-591: ordinal not in range(128)

Surprisingly, it parses nginx even when it is already installed. Is it really necessary? Can't it just renew the installed certificates? When I run

    certbot certonly --nginx --dry-run

it gives the same UnicodeError. Shouldn't it just install the certificate to its own path?
>Is it really necessary?

If you use `--nginx` to renew the certificate, then it needs to be able to load, modify and save your configuration. As a result, the `UnicodeError` will be encountered during renewal.

One possible workaround would be to use `--webroot` rather than `--nginx` to renew.

Another workaround would be to [install Certbot from snap](https://certbot.eff.org/lets-encrypt/centosrhel7-nginx) rather than from CentOS EPEL. This should work because the snap uses Python 3, where (I believe) this issue doesn't exist.
" to use --webroot"  - thanks, I'll try that. Initially I tried to use snap, but it failed because I had no squashfs/fuse on my OpenVZ VPS. So I may just easier write a custom script to apply changes back and forward.

However, I'm a bit surprised that Python 2 is the major language for this bot (at least for OS packages). I very much hope that it will be fixed in Python 2 - do you think it might be done in several months, or will the command just drop Python 2 support (and these OS packages)?

Resolving patch:
diff --git a/certbot-nginx/certbot_nginx/_internal/http_01.py b/certbot-nginx/certbot_nginx/_internal/http_01.py
--- a/certbot-nginx/certbot_nginx/_internal/http_01.py
+++ b/certbot-nginx/certbot_nginx/_internal/http_01.py
@@ -1,5 +1,6 @@
 """A class that performs HTTP-01 challenges for Nginx"""
 
+import io
 import logging
 
 from acme import challenges
@@ -102,7 +103,7 @@ def _mod_config(self):
         self.configurator.reverter.register_file_creation(
             True, self.challenge_conf)
 
-        with open(self.challenge_conf, "w") as new_conf:
+        with io.open(self.challenge_conf, "w", encoding="utf-8") as new_conf:
             nginxparser.dump(config, new_conf)
 
     def _default_listen_addresses(self):
diff --git a/certbot-nginx/certbot_nginx/_internal/nginxparser.py b/certbot-nginx/certbot_nginx/_internal/nginxparser.py
--- a/certbot-nginx/certbot_nginx/_internal/nginxparser.py
+++ b/certbot-nginx/certbot_nginx/_internal/nginxparser.py
@@ -16,6 +16,7 @@
 from pyparsing import White
 from pyparsing import ZeroOrMore
 import six
+from acme.magic_typing import IO, Any # pylint: disable=unused-import
 
 logger = logging.getLogger(__name__)
 
@@ -130,26 +131,27 @@ def load(_file):
 
 
 def dumps(blocks):
-    """Dump to a string.
+    # type: (UnspacedList) -> six.text_type
+    """Dump to a Unicode string.
 
     :param UnspacedList block: The parsed tree
-    :param int indentation: The number of spaces to indent
-    :rtype: str
+    :rtype: six.text_type
 
     """
-    return str(RawNginxDumper(blocks.spaced))
+    return six.text_type(RawNginxDumper(blocks.spaced))
 
 
 def dump(blocks, _file):
+    # type: (UnspacedList, IO[Any]) -> None
     """Dump to a file.
 
     :param UnspacedList block: The parsed tree
-    :param file _file: The file to dump to
-    :param int indentation: The number of spaces to indent
-    :rtype: NoneType
+    :param IO[Any] _file: The file stream to dump to. It must be opened with
+                          Unicode encoding.
+    :rtype: None
 
     """
-    return _file.write(dumps(blocks))
+    _file.write(dumps(blocks))
 
 
 spacey = lambda x: (isinstance(x, six.string_types) and x.isspace()) or x == ''
diff --git a/certbot-nginx/certbot_nginx/_internal/parser.py b/certbot-nginx/certbot_nginx/_internal/parser.py
--- a/certbot-nginx/certbot_nginx/_internal/parser.py
+++ b/certbot-nginx/certbot_nginx/_internal/parser.py
@@ -249,7 +249,7 @@ def filedump(self, ext='tmp', lazy=True):
                     continue
                 out = nginxparser.dumps(tree)
                 logger.debug('Writing nginx conf tree to %s:\n%s', filename, out)
-                with open(filename, 'w') as _file:
+                with io.open(filename, 'w', encoding='utf-8') as _file:
                     _file.write(out)
 
             except IOError:


Test patch:
diff --git a/certbot-ci/certbot_integration_tests/nginx_tests/nginx_config.py b/certbot-ci/certbot_integration_tests/nginx_tests/nginx_config.py
--- a/certbot-ci/certbot_integration_tests/nginx_tests/nginx_config.py
+++ b/certbot-ci/certbot_integration_tests/nginx_tests/nginx_config.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 """General purpose nginx test configuration generator."""
 import getpass
 
@@ -42,6 +43,8 @@ def construct_nginx_config(nginx_root, nginx_webroot, http_port, https_port, oth
   worker_connections 1024;
 }}
 
+# “This comment contains valid Unicode”.
+
 http {{
   # Set an array of temp, cache and log file options that will otherwise default to
   # restricted locations accessible only to root.
@@ -51,61 +54,61 @@ def construct_nginx_config(nginx_root, nginx_webroot, http_port, https_port, oth
   #scgi_temp_path {nginx_root}/scgi_temp;
   #uwsgi_temp_path {nginx_root}/uwsgi_temp;
   access_log {nginx_root}/error.log;
-  
+
   # This should be turned off in a Virtualbox VM, as it can cause some
   # interesting issues with data corruption in delivered files.
   sendfile off;
-  
+
   tcp_nopush on;
   tcp_nodelay on;
   keepalive_timeout 65;
   types_hash_max_size 2048;
-  
+
   #include /etc/nginx/mime.types;
   index index.html index.htm index.php;
-  
+
   log_format   main '$remote_addr - $remote_user [$time_local] $status '
     '"$request" $body_bytes_sent "$http_referer" '
     '"$http_user_agent" "$http_x_forwarded_for"';
-    
+
   default_type application/octet-stream;
-  
+
   server {{
     # IPv4.
     listen {http_port} {default_server};
     # IPv6.
     listen [::]:{http_port} {default_server};
     server_name nginx.{wtf_prefix}.wtf nginx2.{wtf_prefix}.wtf;
-    
+
     root {nginx_webroot};
-    
+
     location / {{
       # First attempt to serve request as file, then as directory, then fall
       # back to index.html.
       try_files $uri $uri/ /index.html;
     }}
   }}
-  
+
   server {{
     listen {http_port};
     listen [::]:{http_port};
     server_name nginx3.{wtf_prefix}.wtf;
-    
+
     root {nginx_webroot};
-    
+
     location /.well-known/ {{
       return 404;
     }}
-    
+
     return 301 https://$host$request_uri;
   }}
-  
+
   server {{
     listen {other_port};
     listen [::]:{other_port};
     server_name nginx4.{wtf_prefix}.wtf nginx5.{wtf_prefix}.wtf;
   }}
-  
+
   server {{
     listen {http_port};
     listen [::]:{http_port};
diff --git a/certbot-nginx/tests/parser_test.py b/certbot-nginx/tests/parser_test.py
--- a/certbot-nginx/tests/parser_test.py
+++ b/certbot-nginx/tests/parser_test.py
@@ -492,6 +492,14 @@ def test_valid_unicode_characters(self):
         self.assertEqual(['server'], parsed[0][2][0])
         self.assertEqual(['listen', '80'], parsed[0][2][1][3])
 
+    def test_valid_unicode_roundtrip(self):
+        """This tests the parser's ability to load and save a config containing Unicode"""
+        nparser = parser.NginxParser(self.config_path)
+        nparser._parse_files(
+            nparser.abs_path('valid_unicode_comments.conf')
+        ) # pylint: disable=protected-access
+        nparser.filedump(lazy=False)
+
     def test_invalid_unicode_characters(self):
         with self.assertLogs() as log:
             nparser = parser.NginxParser(self.config_path)