Repository: gitpython-developers/GitPython

Issue:
Git.execute treats shell=False the same as shell=None
### Expected behavior

The *intention* in [`git.cmd.Git.execute`](https://github.com/gitpython-developers/GitPython/blob/7d4f6c68c657586f27125f25b15f02c5a3d7f35c/git/cmd.py#L827) is that when a `bool` value, `True` or `False`, is passed for `shell`, that value determines whether a shell is used, and that only if this argument is omitted or passed as `None` is the class or object's `USE_SHELL` attribute consulted:

https://github.com/gitpython-developers/GitPython/blob/7d4f6c68c657586f27125f25b15f02c5a3d7f35c/git/cmd.py#L839

https://github.com/gitpython-developers/GitPython/blob/7d4f6c68c657586f27125f25b15f02c5a3d7f35c/git/cmd.py#L903-L905

### The logic error

This is the code [in the `Popen` call](https://github.com/gitpython-developers/GitPython/blob/7d4f6c68c657586f27125f25b15f02c5a3d7f35c/git/cmd.py#L987) that is intended to implement that documented logic:

https://github.com/gitpython-developers/GitPython/blob/7d4f6c68c657586f27125f25b15f02c5a3d7f35c/git/cmd.py#L995

`shell is not None and shell or self.USE_SHELL` is intended to mean `self.USE_SHELL if shell is None else shell` (which could also be written `shell if shell is not None else self.USE_SHELL`). But it actually means `shell or self.USE_SHELL`:

```python
>>> def ternary_truth_table(f):
...     return {(p, q): f(p, q) for p in (True, False, None) for q in (True, False, None)}
...
>>> table1 = ternary_truth_table(lambda p, q: p is not None and p or q)
>>> table2 = ternary_truth_table(lambda p, q: p or q)
>>> table1 == table2
True
>>> table1
{(True, True): True, (True, False): True, (True, None): True, (False, True): True, (False, False): False, (False, None): None, (None, True): True, (None, False): False, (None, None): None}
```

(This resembles the [`type(z)==types.ComplexType and z.real or z`](https://mail.python.org/pipermail/python-dev/2005-September/056510.html) problem that [led to the adoption of conditional expressions](https://peps.python.org/pep-0308/#id5).)

### Possible fix

I would *like* to fix this by doing:

```diff
-                    shell=shell is not None and shell or self.USE_SHELL,
+                    shell=self.USE_SHELL if shell is None else shell,
```

Or, perhaps even more clearly:

```diff
+        if shell is None:
+            shell = self.USE_SHELL
...
-                    shell=shell is not None and shell or self.USE_SHELL,
+                    shell=shell,
```

### Could users be relying on the current (wrong) behavior?

On the surface that seems okay. The problem only happens when `USE_SHELL` is `True`. The class attribute `USE_SHELL` is set to `False` by default. It is named as a constant so, unless documented otherwise, it should really only be set during initialization, or by being monkey-patched inside GitPython's own tests. So it's tempting to think nobody should be relying on what happens if they change it.

Except... setting it from outside GitPython *is* documented. In the changelog entry for [0.3.7](https://github.com/gitpython-developers/GitPython/blob/7d4f6c68c657586f27125f25b15f02c5a3d7f35c/doc/source/changes.rst?plain=1#L669-L684):

https://github.com/gitpython-developers/GitPython/blob/7d4f6c68c657586f27125f25b15f02c5a3d7f35c/doc/source/changes.rst?plain=1#L679-L682

This is [elaborated](https://github.com/gitpython-developers/GitPython/compare/0.3.6...0.3.7#diff-35a18a749eb4d6efad45e56e78a9554926be5526e2ba2159b44311e718450e88) at the `USE_SHELL` class attribute definition:

https://github.com/gitpython-developers/GitPython/blob/7d4f6c68c657586f27125f25b15f02c5a3d7f35c/git/cmd.py#L282-L286

### The `pythonw` use case

In #126, a default of `shell=True` had been introduced to fix a bug on Windows where console windows would flash on the screen for `git` commands, when using the `pythonw` interpreter executable (selected to open `.pyw` files on Windows):

> The GitPython module works great, but when called from a windowless script (.pyw) each command spawns a console window. Those windows are empty and last only a second, but I think they should not be there at all.
>
> I saw it was using `Popen` to call the `git` executable and added a `shell=True` parameter to avoid spawning those windows.

I don't know if that is in practice ever still an issue or not. On my Windows 10 system, I created a Python 3.7.9 virtual environment (since 3.7 is the lowest version GitPython currently supports), installing GitPython in it, running IDLE in it using `pythonw -m idlelib.idle`, and in IDLE, running statements such as `import git` and `git.Git().execute("git version")`. No console window was created. However, it may be that this particular manual test is flawed, as IDLE, being a REPL, must be handling standard streams in a way that most other `.pyw` applications aren't. Nonetheless, the unwanted window creation seems like it should only have happened due to a bug in Python or Windows. But I am not sure.

### Impact of a fix?

My concern is that existing software may be inadvertently relying on `Git.USE_SHELL` taking precedence over the optional `shell` keyword argument to the `Git.execute` method or to any function that forwards that argument to that method.

I believe this should be fixed, because applications that build on GitPython should be able to rely on documented claims about when shells are used to create subprocesses. But I am unsure if there is anything that should be documented or warned about, or taken account in deciding exactly how to fix this and how, if at all, the impact of the fix should be documented in the changelog or elsewhere.

I plan to open a pull request very soon to propose a fix along the lines of those I suggested above. But I wanted to open an issue to explore this, in case that fix is unsatisfactory, requires refinements, or has obscure effects (perhaps even only in cases that GitPython is used incorrectly) that people might find this by searching about.

Comments:

Patch:
diff --git a/git/cmd.py b/git/cmd.py
index 9921dd6c..53b8b905 100644
--- a/git/cmd.py
+++ b/git/cmd.py
@@ -974,6 +974,8 @@ class Git(LazyMixin):
         istream_ok = "None"
         if istream:
             istream_ok = "<valid stream>"
+        if shell is None:
+            shell = self.USE_SHELL
         log.debug(
             "Popen(%s, cwd=%s, universal_newlines=%s, shell=%s, istream=%s)",
             redacted_command,
@@ -992,7 +994,7 @@ class Git(LazyMixin):
                     stdin=istream or DEVNULL,
                     stderr=PIPE,
                     stdout=stdout_sink,
-                    shell=shell is not None and shell or self.USE_SHELL,
+                    shell=shell,
                     close_fds=is_posix,  # unsupported on windows
                     universal_newlines=universal_newlines,
                     creationflags=PROC_CREATIONFLAGS,


Test patch:
diff --git a/test-requirements.txt b/test-requirements.txt
index 1c08c736..9414da09 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -1,6 +1,7 @@
 black
 coverage[toml]
-ddt>=1.1.1, !=1.4.3
+ddt >= 1.1.1, != 1.4.3
+mock ; python_version < "3.8"
 mypy
 pre-commit
 pytest
diff --git a/test/test_git.py b/test/test_git.py
index 48130953..583c74fa 100644
--- a/test/test_git.py
+++ b/test/test_git.py
@@ -4,23 +4,31 @@
 #
 # This module is part of GitPython and is released under
 # the BSD License: https://opensource.org/license/bsd-3-clause/
+import contextlib
+import logging
 import os
+import os.path as osp
+import re
 import shutil
 import subprocess
 import sys
 from tempfile import TemporaryDirectory, TemporaryFile
-from unittest import mock, skipUnless
+from unittest import skipUnless
 
-from git import Git, refresh, GitCommandError, GitCommandNotFound, Repo, cmd
-from test.lib import TestBase, fixture_path
-from test.lib import with_rw_directory
-from git.util import cwd, finalize_process
+if sys.version_info >= (3, 8):
+    from unittest import mock
+else:
+    import mock  # To be able to examine call_args.kwargs on a mock.
 
-import os.path as osp
+import ddt
 
+from git import Git, refresh, GitCommandError, GitCommandNotFound, Repo, cmd
 from git.compat import is_win
+from git.util import cwd, finalize_process
+from test.lib import TestBase, fixture_path, with_rw_directory
 
 
+@ddt.ddt
 class TestGit(TestBase):
     @classmethod
     def setUpClass(cls):
@@ -73,7 +81,50 @@ class TestGit(TestBase):
         res = self.git.transform_kwargs(**{"s": True, "t": True})
         self.assertEqual({"-s", "-t"}, set(res))
 
-    def test_it_executes_git_to_shell_and_returns_result(self):
+    _shell_cases = (
+        # value_in_call, value_from_class, expected_popen_arg
+        (None, False, False),
+        (None, True, True),
+        (False, True, False),
+        (False, False, False),
+        (True, False, True),
+        (True, True, True),
+    )
+
+    def _do_shell_combo(self, value_in_call, value_from_class):
+        with mock.patch.object(Git, "USE_SHELL", value_from_class):
+            # git.cmd gets Popen via a "from" import, so patch it there.
+            with mock.patch.object(cmd, "Popen", wraps=cmd.Popen) as mock_popen:
+                # Use a command with no arguments (besides the program name), so it runs
+                # with or without a shell, on all OSes, with the same effect. Since git
+                # errors out when run with no arguments, we swallow that error.
+                with contextlib.suppress(GitCommandError):
+                    self.git.execute(["git"], shell=value_in_call)
+
+        return mock_popen
+
+    @ddt.idata(_shell_cases)
+    def test_it_uses_shell_or_not_as_specified(self, case):
+        """A bool passed as ``shell=`` takes precedence over `Git.USE_SHELL`."""
+        value_in_call, value_from_class, expected_popen_arg = case
+        mock_popen = self._do_shell_combo(value_in_call, value_from_class)
+        mock_popen.assert_called_once()
+        self.assertIs(mock_popen.call_args.kwargs["shell"], expected_popen_arg)
+
+    @ddt.idata(full_case[:2] for full_case in _shell_cases)
+    def test_it_logs_if_it_uses_a_shell(self, case):
+        """``shell=`` in the log message agrees with what is passed to `Popen`."""
+        value_in_call, value_from_class = case
+
+        with self.assertLogs(cmd.log, level=logging.DEBUG) as log_watcher:
+            mock_popen = self._do_shell_combo(value_in_call, value_from_class)
+
+        popen_shell_arg = mock_popen.call_args.kwargs["shell"]
+        expected_message = re.compile(rf"DEBUG:git.cmd:Popen\(.*\bshell={popen_shell_arg}\b.*\)")
+        match_attempts = [expected_message.fullmatch(message) for message in log_watcher.output]
+        self.assertTrue(any(match_attempts), repr(log_watcher.output))
+
+    def test_it_executes_git_and_returns_result(self):
         self.assertRegex(self.git.execute(["git", "version"]), r"^git version [\d\.]{2}.*$")
 
     def test_it_executes_git_not_from_cwd(self):