Repository: pallets/click

Problem statement:
Resource tracker warning on fish completion
Issue:
I am having the same issue described in #1738 although I am using `fish`. Essentially, I get an error from `resource_tracker.py` related to "leaked semaphore objects," when I run the completion. The completion still works, but the text of the error is garbled along with the completion information before tabbing between options. This does not happen when I use Python 3.7, but does when I use Python 3.8.

The error I am getting ends up looking like this after pressing `tab` to get completion text (the third line starting with `❯` is where I pressed `tab`):

```
----
❯ poetry shell
Spawning shell within ----.p3d-sRVjZJIG-py3.8
source ----.p3d-sRVjZJIG-py3.8/bin/activate.fish
Welcome to fish, the friendly interactive shell

----
❯ source ----.p3d-sRVjZJIG-py3.8/bin/activate.fish

----
❯ p3d /Users/----/.asdf/installs/python/3.8.8/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 32 leaked semaphore objects to clean up at shutdownz API calls.)  pipeline  (Run pipeline processes.)
  warnings.warn('resource_tracker: There appear to be %d '

```

Reproduce:
I don't have a cut down example that I can provide, but I am wondering if maybe the fact that I am using `poetry` and trying to run completion in a virtualenv is causing an issue? As noted above, this works for Python 3.7 but has an issue with 3.8.

I am also using `click-didyoumean` and I manage my Python installations using `asdf`.

Expected Behavior:
I would expect the same behavior (no error) across different supported versions of Python.

Environment:

- Python version: 3.8.8
- OS: macOS 10.15.7
- Click version: 8.0.1
- click-didyoumean version: 0.0.3
- asdf version: 0.8.1
- Poetry version: 1.1.7



Hints:
Closing for the same reason as the linked issue. I can't reproduce this. This does not appear to be an issue with Click, but with something else about your project.
@davidism So, I have been trying to track down this error further. I am talking with @Delgan who is the dev for `loguru` which appeared to be where the error was coming from (this is our discussion so far: https://github.com/Delgan/loguru/issues/477). We have actually been able to narrow it down to the use of `multiprocessing.SimpleQueue()` on macOS.

This simple test produces the error reliably for me:
`cli.py`:
```python
#!/usr/bin/env python
# -*- coding: utf-8 -*-


"""Simple example of issue caused by click completion + multiprocessing.SimpleQueue()."""


####################################################################################################
# Imports
####################################################################################################


# ==Standard Library==
import multiprocessing

# ==Site-Packages==
import click


####################################################################################################
# Trigger Issue
####################################################################################################


my_queue = multiprocessing.SimpleQueue()


####################################################################################################
# Main
####################################################################################################


@click.command()
def sub_a() -> None:
    """Subcommand A."""

    print("A")


@click.command()
def sub_b() -> None:
    """Subcommand B."""

    print("B")


@click.group("tool")
def command() -> None:
    """Root command group."""

    pass


# ------------------------------------------------------------------------------------------------ #


# Assemble subcommands
command.add_command(sub_a)
command.add_command(sub_b)
```

`pyproject.toml`:
```toml
[tool.poetry]
name = "tool"
version = "1.0.0"
description = ""
authors = ["---- <---->"]
packages = [
    {include = "tool", from = "src"},
]

[tool.poetry.dependencies]
python = "^3.7"

click = "^8.0.1"

[tool.poetry.dev-dependencies]

[tool.poetry.scripts]
tool = "tool.cli:command"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
```

`tool.fish`:
```fish
#!/usr/bin/env fish

function _tool_completion;
    set -l response;

    for value in (env _TOOL_COMPLETE=fish_complete COMP_WORDS=(commandline -cp) COMP_CWORD=(commandline -t) tool);
        set response $response $value;
    end;

    for completion in $response;
        set -l metadata (string split "," $completion);

        if test $metadata[1] = "dir";
            __fish_complete_directories $metadata[2];
        else if test $metadata[1] = "file";
            __fish_complete_path $metadata[2];
        else if test $metadata[1] = "plain";
            echo $metadata[2];
        end;
    end;
end;

complete --no-files --command tool --arguments "(_tool_completion)";
```
The error only shows up when I try to run the completion. If I run `cli.py` directly, there are no errors.
Try replacing this line with `sys.exit()`, that's the only meaningful difference between normal and completion mode that I can think of: https://github.com/pallets/click/blob/ddcabdd4f22f438c0c963150b930d0d09b04dea7/src/click/core.py#L1133
@davidism Hey, that fixed the issue. I can get the error to trigger just by running this simple script:
```python
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import multiprocessing
import os
import sys


my_queue = multiprocessing.SimpleQueue()

os._exit(0)
```
The issue definitely appears to be due to some sort of resources not being freed when using `os._exit()` as opposed to the usual `sys.exit()`. Seems like it may have something to do with this (from the `multiprocessing` docs:

> Contexts and start methods
> Depending on the platform, multiprocessing supports three ways to start a process. These start methods are
> 
> spawn
> The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process object’s run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.
> 
> Available on Unix and Windows. The default on Windows and macOS.
> 
> fork
> The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.
> 
> Available on Unix only. The default on Unix.
> 
> forkserver
> When the program starts and selects the forkserver start method, a server process is started. From then on, whenever a new process is needed, the parent process connects to the server and requests that it fork a new process. The fork server process is single threaded so it is safe for it to use os.fork(). No unnecessary resources are inherited.
> 
> Available on Unix platforms which support passing file descriptors over Unix pipes.


Also, this seems to confirm it:

> os._exit(n)
> Exit the process with status n, without calling cleanup handlers, flushing stdio buffers, etc.
> 
> Note The standard way to exit is sys.exit(n). _exit() should normally only be used in the child process after a fork().
I feel like we've discussed whether `_fast_exit` was good before, but I can't find the discussion. It sounds like we should remove it, it's bypassing Python cleanup for potentially slightly faster completion responses. Would you mind submitting a PR?

Patch:
diff --git a/src/click/core.py b/src/click/core.py
--- a/src/click/core.py
+++ b/src/click/core.py
@@ -46,17 +46,6 @@
 V = t.TypeVar("V")
 
 
-def _fast_exit(code: int) -> "te.NoReturn":
-    """Low-level exit that skips Python's cleanup but speeds up exit by
-    about 10ms for things like shell completion.
-
-    :param code: Exit code.
-    """
-    sys.stdout.flush()
-    sys.stderr.flush()
-    os._exit(code)
-
-
 def _complete_visible_commands(
     ctx: "Context", incomplete: str
 ) -> t.Iterator[t.Tuple[str, "Command"]]:
@@ -1130,7 +1119,7 @@ def _main_shell_completion(
         from .shell_completion import shell_complete
 
         rv = shell_complete(self, ctx_args, prog_name, complete_var, instruction)
-        _fast_exit(rv)
+        sys.exit(rv)
 
     def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
         """Alias for :meth:`main`."""


Test patch:
diff --git a/tests/test_shell_completion.py b/tests/test_shell_completion.py
--- a/tests/test_shell_completion.py
+++ b/tests/test_shell_completion.py
@@ -1,5 +1,3 @@
-import sys
-
 import pytest
 
 from click.core import Argument
@@ -265,7 +263,6 @@ def test_completion_item_data():
 
 @pytest.fixture()
 def _patch_for_completion(monkeypatch):
-    monkeypatch.setattr("click.core._fast_exit", sys.exit)
     monkeypatch.setattr(
         "click.shell_completion.BashComplete._check_version", lambda self: True
     )