Repository: certbot/certbot

Issue:
Windows: Renewal deploy hook opens in notepad.exe instead of being executed
I placed a .ps1 Powershell script into `C:\Certbot\renewal-hooks\deploy` but Certbot does not execute it. Instead, it opens the script with Windows Notepad.

## My operating system is (include version):
Windows 10 1809

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):
https://dl.eff.org/certbot-beta-installer-win32.exe

## I ran this command and it produced this output:
`certbot renew --force-renewal`

```
Saving debug log to C:\Certbot\log\letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing C:\Certbot\renewal\miketest.pwsd76.ab.ca.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator standalone, Installer None
Renewing an existing certificate
Running deploy-hook command: C:\Certbot\renewal-hooks\deploy\LM-CertUpload.ps1
```
Then Windows' notepad.exe opened up, showing my .ps1 script.

## Certbot's behavior differed from what I expected because:
The [forum post about the Certbot Windows beta](https://community.letsencrypt.org/t/beta-phase-of-certbot-for-windows/105822) explicitly states that .ps1 hooks are supposed to work:
> Similarly to Linux, Certbot for Windows supports any file executable for this platform, if it is available in the PATH. It can be .bat, .ps1 or even executable binaries like .exe.


Discussion:
**TL;DR**

Sadly for now Powershell scripts will not work in the `renewal-hooks` folder. Only BAT files or executable binaries will.

If you really want to use Powershell scripts, explicitly pass the executable interpreter of the script to the relevant CLI hook flag. For instance, with a Powershell script, use `--deploy-hook "powershell.exe -File C:\path\to\your\script.ps1"`. Alternatively create a BAT file, that will be your hook in `renewal-hooks`, and put the `powershell.exe -File ... ` in it. 

**Complete explanation**

On every OS system (Linux, Windows, ...), you run something by passing a file to one entrypoint of the `exec` family supported by the kernel (`system_call`, `CreateProcess` ...). Only pre-compiled executable binaries are accepted. Scripts in particular are only a bunch of text that have no meaning for the kernel, and would lead to a runtime error.

However, Linux has a very nice feature called the shebang, which consists in writing as the first line of the script a description of the binaries executable (the interpreter) to use to read the script (eg. `#!/bin/bash`). This shebang is intercepted by the kernel on the exec entrypoint, and will invoke the described interpreter and pass the script to it. This allows you to "execute" seamlessly a script on Linux.

There is no such mechanism on Windows. The "best thing" you have is the association of a program to a kind of file to read it, the "Opens with" feature. But it is a user space functionality, not a kernel one, and its behavior depends totally on the program that implements it. Primary implemented use is when you double-click the file. Secondary implemented use is when you type the name of the file in a CMD shell. In this second situation, given that the `*.py` type of files is associated to `python.exe`, if you type:
```
.\test.py
```

The CMD shell will translated that into:
```
python.exe .\test.py
```

Which effectively run the python script.

Note that for the CMD shell, `*.bat` are the native scripts for this shell, so:
```
.\test.bat
```

Is also working out-of-the-box.

Now, what Python does of all of this in the `subprocess` module ?
* `subprocess.call(['my_executable', 'param1']) will directly pass to the `CreateProcess` method of the kernel. So as said before and whatever you do, you cannot use scripts here, only executable binaries
* `subprocess.call('my_executable param1', shell=True)` will open a CMD shell, and pass the command string to it. Here, the "Open with" feature can apply.

For several reasons, Certbot uses `subprocess.call('my_executable param1', shell=True)` to run hook scripts. So, we can have these situations:
* if you pass an executable binary, it will work directly
* if you pass a `.bat` file, it will work also, because CMD shell because BAT files are native scripts for CMD
* if you pass a `.py` file, it will work also, because Python for Windows associate `*.py` with `python.exe`, and `python.exe script.py` as run by CMD in this case, is effectively running the script
 
And finally ... `.ps1` files will not work ! Because:
* powershell scripts are not native for CMD, so it relies on the "Open with" feature
* extremely unfortunately, Notepad is the default program associated to `*.ps1` files, and so the result will be to have Notepad opened with the content of the script, as observed in this issue.
* extremely stupidly, even if you associate `powershell.exe` to `*.ps1` files, it will not work either. Indeed, on the contrary to Python interpreter where you pass the script as the first parameter, for Powershell, it is `powershell.exe -File test.ps1`. The command `powershell.exe test.ps1`, as created by CMD during the subprocess call, is invalid ...

Finally, the sane way to run hooks is what I said in the TL;DR: pass the full command of `interpreter + [whatever flags needed] + your_script`. This will work in the CMD shell, because you are passing a binary executable, and so it will work as well for the Certbot hooks.

**What can be improved ?**

Arguably passing Powershell scripts as Certbot hooks is a decent choice on Windows. The two other choices are BAT files and Python files. We need to improve Certbot hooks for Windows here, by appending the correct interpreter. This is mandatory for scripts in `renewal-hooks`, and a nice feature for the CLI hooks flags. Given the script extension, and if a `interpreter + [whatever flags needed] + your_script` is not provided on the CLI / script is in `renewal-hooks`, we would have the following behavior:
* For BAT files, it works out-of-the-box, no interpreter is needed
* For Python files, we could append the `sys.executable`, to make these scripts runnable even if Python is not installed globally and the system (and so the "Open with" is not defined for `.py` files)
* For Powershell scripts, appending `powershell.exe -File` would do the trick.

I keep the issue opened until we have a decision about what we do, or we do not. @bmw, what are your thoughts here ?

In the meantime, would it be possible for somebody to edit [the forum post about the Certbot Windows beta](https://community.letsencrypt.org/t/beta-phase-of-certbot-for-windows/105822)? AFAIK that post is the closest thing to official release notes for the Windows version, and it explicitly states that .ps1 hooks work:

> Similarly to Linux, Certbot for Windows supports any file executable for this platform, if it is available in the PATH. It can be .bat, .ps1 or even executable binaries like .exe.

Obviously incorrect.
I updated the text to say:

> Certbot for Windows supports any file that is executable on the Windows Command Prompt, if it is available in your PATH. It’s important to note that Powershell scripts do not currently work because they cannot be run directly from the Windows Command Prompt and instead they must be given as arguments to powershell.exe. If you would like to use a Powershell script, you should probably wrap it in something like a .bat file that properly passes it to Powershell.

@mikenor, thanks for taking the time to report this issue.

@adferrand, this is a pretty unfortunate problem. I'm a little hesitant to do much mangling of the hooks given to Certbot. Introducing differences between the command/executable given to Certbot and what we actually run to me seems like it's an area ripe with potential bugs and confusion.

Trying to get into the mind of a Windows sysadmin, I wonder if they'd consider a Powershell script as an executable file. Windows seems to go out of their way to not make it one with the behavior you described above. Without us advertising that Certbot will execute `.ps1` files, would a Windows sysadmin have tried it expecting it to work? If not, I don't think we should do anything here. Given all of the things we have to work on in Certbot, I personally think one reasonable option for us is to hold off on making a change and see if others hit this issue or request a similar change.

If other people do hit this issue or we're confident that people would expect a Powershell script to work, my preference would be if we're on Windows and the file ends in `.ps1`, fix up the command to run the script with Powershell. I personally don't think we should make any other changes such as changing the Python interpreter used as I think it's just as likely if not more so that the user wants to use the system Python with their own dependencies as our own copy of Python installed specifically for Certbot.

Resolving patch:
diff --git a/certbot/certbot/_internal/hooks.py b/certbot/certbot/_internal/hooks.py
--- a/certbot/certbot/_internal/hooks.py
+++ b/certbot/certbot/_internal/hooks.py
@@ -2,14 +2,13 @@
 from __future__ import print_function
 
 import logging
-from subprocess import PIPE
-from subprocess import Popen
 
 from acme.magic_typing import List
 from acme.magic_typing import Set
 from certbot import errors
 from certbot import util
 from certbot.compat import filesystem
+from certbot.compat import misc
 from certbot.compat import os
 from certbot.plugins import util as plug_util
 
@@ -229,36 +228,10 @@ def _run_hook(cmd_name, shell_cmd):
     :type shell_cmd: `list` of `str` or `str`
 
     :returns: stderr if there was any"""
-    err, _ = execute(cmd_name, shell_cmd)
+    err, _ = misc.execute_command(cmd_name, shell_cmd)
     return err
 
 
-def execute(cmd_name, shell_cmd):
-    """Run a command.
-
-    :param str cmd_name: the user facing name of the hook being run
-    :param shell_cmd: shell command to execute
-    :type shell_cmd: `list` of `str` or `str`
-
-    :returns: `tuple` (`str` stderr, `str` stdout)"""
-    logger.info("Running %s command: %s", cmd_name, shell_cmd)
-
-    # universal_newlines causes Popen.communicate()
-    # to return str objects instead of bytes in Python 3
-    cmd = Popen(shell_cmd, shell=True, stdout=PIPE,
-                stderr=PIPE, universal_newlines=True)
-    out, err = cmd.communicate()
-    base_cmd = os.path.basename(shell_cmd.split(None, 1)[0])
-    if out:
-        logger.info('Output from %s command %s:\n%s', cmd_name, base_cmd, out)
-    if cmd.returncode != 0:
-        logger.error('%s command "%s" returned error code %d',
-                     cmd_name, shell_cmd, cmd.returncode)
-    if err:
-        logger.error('Error output from %s command %s:\n%s', cmd_name, base_cmd, err)
-    return err, out
-
-
 def list_hooks(dir_path):
     """List paths to all hooks found in dir_path in sorted order.
 
diff --git a/certbot/certbot/_internal/plugins/manual.py b/certbot/certbot/_internal/plugins/manual.py
--- a/certbot/certbot/_internal/plugins/manual.py
+++ b/certbot/certbot/_internal/plugins/manual.py
@@ -9,6 +9,7 @@
 from certbot import interfaces
 from certbot import reverter
 from certbot._internal import hooks
+from certbot.compat import misc
 from certbot.compat import os
 from certbot.plugins import common
 
@@ -186,4 +187,4 @@ def cleanup(self, achalls):  # pylint: disable=missing-function-docstring
         self.reverter.recovery_routine()
 
     def _execute_hook(self, hook_name):
-        return hooks.execute(self.option_name(hook_name), self.conf(hook_name))
+        return misc.execute_command(self.option_name(hook_name), self.conf(hook_name))
diff --git a/certbot/certbot/compat/misc.py b/certbot/certbot/compat/misc.py
--- a/certbot/certbot/compat/misc.py
+++ b/certbot/certbot/compat/misc.py
@@ -4,12 +4,16 @@
 """
 from __future__ import absolute_import
 
+import logging
 import select
+import subprocess
 import sys
 
 from certbot import errors
 from certbot.compat import os
 
+from acme.magic_typing import Tuple
+
 try:
     from win32com.shell import shell as shellwin32
     POSIX_MODE = False
@@ -17,6 +21,7 @@
     POSIX_MODE = True
 
 
+logger = logging.getLogger(__name__)
 
 # For Linux: define OS specific standard binary directories
 STANDARD_BINARY_DIRS = ["/usr/sbin", "/usr/local/bin", "/usr/local/sbin"] if POSIX_MODE else []
@@ -109,3 +114,39 @@ def underscores_for_unsupported_characters_in_path(path):
     # Windows specific
     drive, tail = os.path.splitdrive(path)
     return drive + tail.replace(':', '_')
+
+
+def execute_command(cmd_name, shell_cmd):
+    # type: (str, str) -> Tuple[str, str]
+    """
+    Run a command:
+        - on Linux command will be run by the standard shell selected with Popen(shell=True)
+        - on Windows command will be run in a Powershell shell
+
+    :param str cmd_name: the user facing name of the hook being run
+    :param str shell_cmd: shell command to execute
+
+    :returns: `tuple` (`str` stderr, `str` stdout)
+    """
+    logger.info("Running %s command: %s", cmd_name, shell_cmd)
+
+    if POSIX_MODE:
+        cmd = subprocess.Popen(shell_cmd, shell=True, stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE, universal_newlines=True)
+    else:
+        line = ['powershell.exe', '-Command', shell_cmd]
+        cmd = subprocess.Popen(line, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+                               universal_newlines=True)
+
+    # universal_newlines causes Popen.communicate()
+    # to return str objects instead of bytes in Python 3
+    out, err = cmd.communicate()
+    base_cmd = os.path.basename(shell_cmd.split(None, 1)[0])
+    if out:
+        logger.info('Output from %s command %s:\n%s', cmd_name, base_cmd, out)
+    if cmd.returncode != 0:
+        logger.error('%s command "%s" returned error code %d',
+                     cmd_name, shell_cmd, cmd.returncode)
+    if err:
+        logger.error('Error output from %s command %s:\n%s', cmd_name, base_cmd, err)
+    return err, out


Test patch:
diff --git a/certbot-ci/certbot_integration_tests/assets/hook.py b/certbot-ci/certbot_integration_tests/assets/hook.py
--- a/certbot-ci/certbot_integration_tests/assets/hook.py
+++ b/certbot-ci/certbot_integration_tests/assets/hook.py
@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+from __future__ import print_function
 import os
 import sys
 
@@ -7,5 +8,4 @@
     sys.stderr.write('Environment variables not properly set!\n')
     sys.exit(1)
 
-with open(sys.argv[2], 'a') as file_h:
-    file_h.write(hook_script_type + '\n')
+print(hook_script_type)
diff --git a/certbot-ci/certbot_integration_tests/certbot_tests/assertions.py b/certbot-ci/certbot_integration_tests/certbot_tests/assertions.py
--- a/certbot-ci/certbot_integration_tests/certbot_tests/assertions.py
+++ b/certbot-ci/certbot_integration_tests/certbot_tests/assertions.py
@@ -1,4 +1,5 @@
 """This module contains advanced assertions for the certbot integration tests."""
+import io
 import os
 
 try:
@@ -21,7 +22,8 @@ def assert_hook_execution(probe_path, probe_content):
     :param probe_path: path to the file that received the hook output
     :param probe_content: content expected when the hook is executed
     """
-    with open(probe_path, 'r') as file:
+    encoding = 'utf-8' if POSIX_MODE else 'utf-16'
+    with io.open(probe_path, 'rt', encoding=encoding) as file:
         data = file.read()
 
     lines = [line.strip() for line in data.splitlines()]
diff --git a/certbot-ci/certbot_integration_tests/utils/misc.py b/certbot-ci/certbot_integration_tests/utils/misc.py
--- a/certbot-ci/certbot_integration_tests/utils/misc.py
+++ b/certbot-ci/certbot_integration_tests/utils/misc.py
@@ -140,13 +140,12 @@ def generate_test_file_hooks(config_dir, hook_probe):
             entrypoint_script = '''\
 #!/usr/bin/env bash
 set -e
-"{0}" "{1}" "{2}" "{3}"
+"{0}" "{1}" "{2}" >> "{3}"
 '''.format(sys.executable, hook_path, entrypoint_script_path, hook_probe)
         else:
-            entrypoint_script_path = os.path.join(hook_dir, 'entrypoint.bat')
+            entrypoint_script_path = os.path.join(hook_dir, 'entrypoint.ps1')
             entrypoint_script = '''\
-@echo off
-"{0}" "{1}" "{2}" "{3}"
+& "{0}" "{1}" "{2}" >> "{3}"
             '''.format(sys.executable, hook_path, entrypoint_script_path, hook_probe)
 
         with open(entrypoint_script_path, 'w') as file_h:
diff --git a/certbot/tests/compat/misc_test.py b/certbot/tests/compat/misc_test.py
new file mode 100644
--- /dev/null
+++ b/certbot/tests/compat/misc_test.py
@@ -0,0 +1,48 @@
+"""Tests for certbot.compat.misc"""
+try:
+    import mock
+except ImportError:  # pragma: no cover
+    from unittest import mock  # type: ignore
+import unittest
+
+from certbot.compat import os
+
+
+class ExecuteTest(unittest.TestCase):
+    """Tests for certbot.compat.misc.execute_command."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.compat.misc import execute_command
+        return execute_command(*args, **kwargs)
+
+    def test_it(self):
+        for returncode in range(0, 2):
+            for stdout in ("", "Hello World!",):
+                for stderr in ("", "Goodbye Cruel World!"):
+                    self._test_common(returncode, stdout, stderr)
+
+    def _test_common(self, returncode, stdout, stderr):
+        given_command = "foo"
+        given_name = "foo-hook"
+        with mock.patch("certbot.compat.misc.subprocess.Popen") as mock_popen:
+            mock_popen.return_value.communicate.return_value = (stdout, stderr)
+            mock_popen.return_value.returncode = returncode
+            with mock.patch("certbot.compat.misc.logger") as mock_logger:
+                self.assertEqual(self._call(given_name, given_command), (stderr, stdout))
+
+        executed_command = mock_popen.call_args[1].get(
+            "args", mock_popen.call_args[0][0])
+        if os.name == 'nt':
+            expected_command = ['powershell.exe', '-Command', given_command]
+        else:
+            expected_command = given_command
+        self.assertEqual(executed_command, expected_command)
+
+        mock_logger.info.assert_any_call("Running %s command: %s",
+                                         given_name, given_command)
+        if stdout:
+            mock_logger.info.assert_any_call(mock.ANY, mock.ANY,
+                                             mock.ANY, stdout)
+        if stderr or returncode:
+            self.assertTrue(mock_logger.error.called)
diff --git a/certbot/tests/hook_test.py b/certbot/tests/hook_test.py
--- a/certbot/tests/hook_test.py
+++ b/certbot/tests/hook_test.py
@@ -72,13 +72,13 @@ def _call(cls, *args, **kwargs):  # pragma: no cover
 
     @classmethod
     def _call_with_mock_execute(cls, *args, **kwargs):
-        """Calls self._call after mocking out certbot._internal.hooks.execute.
+        """Calls self._call after mocking out certbot.compat.misc.execute_command.
 
         The mock execute object is returned rather than the return value
         of self._call.
 
         """
-        with mock.patch("certbot._internal.hooks.execute") as mock_execute:
+        with mock.patch("certbot.compat.misc.execute_command") as mock_execute:
             mock_execute.return_value = ("", "")
             cls._call(*args, **kwargs)
         return mock_execute
@@ -292,7 +292,7 @@ class RenewalHookTest(HookTest):
     # pylint: disable=abstract-method
 
     def _call_with_mock_execute(self, *args, **kwargs):
-        """Calls self._call after mocking out certbot._internal.hooks.execute.
+        """Calls self._call after mocking out certbot.compat.misc.execute_command.
 
         The mock execute object is returned rather than the return value
         of self._call. The mock execute object asserts that environment
@@ -313,7 +313,7 @@ def execute_side_effect(*unused_args, **unused_kwargs):
             self.assertEqual(os.environ["RENEWED_LINEAGE"], lineage)
             return ("", "")
 
-        with mock.patch("certbot._internal.hooks.execute") as mock_execute:
+        with mock.patch("certbot.compat.misc.execute_command") as mock_execute:
             mock_execute.side_effect = execute_side_effect
             self._call(*args, **kwargs)
         return mock_execute
@@ -418,42 +418,6 @@ def test_no_overlap(self):
         mock_execute.assert_called_with("deploy-hook", self.config.renew_hook)
 
 
-class ExecuteTest(unittest.TestCase):
-    """Tests for certbot._internal.hooks.execute."""
-
-    @classmethod
-    def _call(cls, *args, **kwargs):
-        from certbot._internal.hooks import execute
-        return execute(*args, **kwargs)
-
-    def test_it(self):
-        for returncode in range(0, 2):
-            for stdout in ("", "Hello World!",):
-                for stderr in ("", "Goodbye Cruel World!"):
-                    self._test_common(returncode, stdout, stderr)
-
-    def _test_common(self, returncode, stdout, stderr):
-        given_command = "foo"
-        given_name = "foo-hook"
-        with mock.patch("certbot._internal.hooks.Popen") as mock_popen:
-            mock_popen.return_value.communicate.return_value = (stdout, stderr)
-            mock_popen.return_value.returncode = returncode
-            with mock.patch("certbot._internal.hooks.logger") as mock_logger:
-                self.assertEqual(self._call(given_name, given_command), (stderr, stdout))
-
-        executed_command = mock_popen.call_args[1].get(
-            "args", mock_popen.call_args[0][0])
-        self.assertEqual(executed_command, given_command)
-
-        mock_logger.info.assert_any_call("Running %s command: %s",
-                                         given_name, given_command)
-        if stdout:
-            mock_logger.info.assert_any_call(mock.ANY, mock.ANY,
-                                             mock.ANY, stdout)
-        if stderr or returncode:
-            self.assertTrue(mock_logger.error.called)
-
-
 class ListHooksTest(test_util.TempDirTestCase):
     """Tests for certbot._internal.hooks.list_hooks."""