Repository: certbot/certbot

Issue:
Using the Nginx plugin when Nginx is turned off hangs forever?
## My operating system is (include version):

Arch Linux

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):

OS package manager

## I ran this command and it produced this output:

`certbot --nginx -vv -d domain.net -d blog.domain.net -d cf.domain.net -d mail.domain.net`

## Certbot's behavior differed from what I expected because:

It hung for 5 - 10 minutes without printing any output.

## Here is a Certbot log showing the issue (if available):
###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

http://pastebin.com/w29rj9F7

## Other info:
Nginx wasn't running when running Certbot. Nginx version is 1.11.10. We should investigate this issue before unhiding Nginx. If we struggle to reproduce, we can kick/close it.

cc @ohemorange 


Discussion:
Yup, I was able to repro. Checking now if it's the distro or the nginx version (that version is mainline, not stable).
1.11.10 works on ubuntu.
Yup, we hang in arch on stable nginx.
This happens when you install from master, so it's not a packaging issue.
On arch (and not ubuntu), `nginx` from the command line returns immediately, but using the `subprocess` library, `nginx` only returns from `communicate` when given an explicit `nginx -s stop`.
Well. We can fix this by not trying to access `stdout` and `stderr`. Which could mean it's some sort of buffering issue in nginx.
Well. We could fix this by `calling subprocess.Popen([nginx_ctl, "-c", nginx_conf]).communicate()` right before we call `reload`. Because then:

1. problem with the config: fail to start, which returns with an error we ignore. fail to reload, which gives us a returncode, so we fail to start again, and then time print the error. all good!
2. was running before: fail to start, which returns with an error we ignore (does slow it down a little because it tries to start a few times). reload successfully. all good!
3. wasn't running before: start successfully, which returns without an error but we still ignore it. reload successfully. all good!
@bmw would that be too disgusting of a solution
relevant code: https://github.com/certbot/certbot/blob/master/certbot-nginx/certbot_nginx/configurator.py#L821
lol [this person](https://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/) agrees:

> A closer inspection of the subprocess.Popen docs revealed a warning "Note: The data read is buffered in memory, so do not use this method if the data size is large or unlimited." I'd probably read that before and assumed that it was a warning about possibly consuming a lot of memory and being inefficient if you try to pass a lot of data around. So I ignored it. The STDOUT chatter of shell scripts that I was collecting for logging purposes did not strike me as "large" (65K is positively tiny these days) and it isn't an application where I'm particularly concerned about memory consumption or efficiency.
> 
> Apparently, that warning actually means "Note: if there's any chance that the data read will be more than a couple pages, this will deadlock your code." What was happening was that the memory buffer was a fixed size. When it filled up, it simply stopped letting the child process write to it. The child would then sit and patiently wait to be able to write the rest of its output.
there is a maybe fix though that's less hacky than this
actually nope that wouldn't solve this
ok we don't really want to call start before reload. but we only need the stdout and stderr if it failed to start. so! there's not really a problem putting the `nginx` call inside the "if we fail to reload", I think.

so let's see:
1. problem with the config: fail to reload, which gives us a bad returncode. fail to start, which returns with an error we ignore. we test the returncode, which says it failed, so we try to start again, and this time when it fails we print the error. (it terminates on failure to start.) all good!
2. was running before: reload successfully. all good!
3. wasn't running before but could be: fail to reload, which gives us a bad returncode. start successfully, which returns without an error, and we don't catch the output but that's fine because it went well. we test the returncode, which says everything is fine, so we move on. all good!

so, we only try to catch the output if we know we've failed to start. which is rare enough that it's fine to call it twice there.

Resolving patch:
diff --git a/certbot-nginx/certbot_nginx/configurator.py b/certbot-nginx/certbot_nginx/configurator.py
--- a/certbot-nginx/certbot_nginx/configurator.py
+++ b/certbot-nginx/certbot_nginx/configurator.py
@@ -5,6 +5,7 @@
 import shutil
 import socket
 import subprocess
+import tempfile
 import time
 
 import OpenSSL
@@ -829,22 +830,22 @@ def nginx_restart(nginx_ctl, nginx_conf="/etc/nginx.conf"):
 
     """
     try:
-        proc = subprocess.Popen([nginx_ctl, "-c", nginx_conf, "-s", "reload"],
-                                stdout=subprocess.PIPE,
-                                stderr=subprocess.PIPE)
-        stdout, stderr = proc.communicate()
+        proc = subprocess.Popen([nginx_ctl, "-c", nginx_conf, "-s", "reload"])
+        proc.communicate()
 
         if proc.returncode != 0:
             # Maybe Nginx isn't running
-            nginx_proc = subprocess.Popen([nginx_ctl, "-c", nginx_conf],
-                                          stdout=subprocess.PIPE,
-                                          stderr=subprocess.PIPE)
-            stdout, stderr = nginx_proc.communicate()
-
-            if nginx_proc.returncode != 0:
-                # Enter recovery routine...
-                raise errors.MisconfigurationError(
-                    "nginx restart failed:\n%s\n%s" % (stdout, stderr))
+            # Write to temporary files instead of piping because of communication issues on Arch
+            # https://github.com/certbot/certbot/issues/4324
+            with tempfile.TemporaryFile() as out:
+                with tempfile.TemporaryFile() as err:
+                    nginx_proc = subprocess.Popen([nginx_ctl, "-c", nginx_conf],
+                        stdout=out, stderr=err)
+                    nginx_proc.communicate()
+                    if nginx_proc.returncode != 0:
+                        # Enter recovery routine...
+                        raise errors.MisconfigurationError(
+                            "nginx restart failed:\n%s\n%s" % (out.read(), err.read()))
 
     except (OSError, ValueError):
         raise errors.MisconfigurationError("nginx restart failed")


Test patch: