Repository: PrefectHQ/prefect

Issue:
Windows: Cancelling process flow run kills agent too
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

I have a prefect agent running in a windows command prompt.
I also have a separate command prompt running prefect orion

If I have a scheduled or manually submitted flow run and I press cancel in Prefect Orion the job correctly transitions to a cancelling state. 

This causes the prefect agent to kill the processes for the flow run however this also sends a kill signal to the agent process itself:

![image](https://user-images.githubusercontent.com/48279914/205299532-65375110-6908-41fa-9e00-1b887ad0923d.png)


Note "Terminate Batch Job Y/N" was not caused by me pressing ctrl+c - this happened from the external signal via prefect orion.

Restarting the agent leaves it in a confused state since it never logs the successful kill of the flow:

![image](https://user-images.githubusercontent.com/48279914/205299503-4bb85de3-d736-4497-85a6-09eef2aa3c7d.png)

This may be linked to issue #7736 


### Reproduction

```python3
Windows, have an agent running, prefect orion running and a flow running.

Cancel a flow run from the UI.

Agent process receives kill signal, locking it at a "Terminate Batch Job Y/N" state.
```


### Error

```python3
After restarting the agent:

14:00:59.727 | ERROR   | prefect.agent - Encountered exception while killing infrastructure for flow run '2c154be8-e775-413a-afa1-19fc994084f9'. Flow run may not be cancelled.
Traceback (most recent call last):
  File "D:\prefect\prefect_agent\lib\site-packages\prefect\agent.py", line 283, in cancel_run
    await infrastructure.kill(flow_run.infrastructure_pid)
  File "D:\prefect\prefect_agent\lib\site-packages\prefect\infrastructure\process.py", line 159, in kill
    os.kill(pid, signal.CTRL_BREAK_EVENT)
OSError: [WinError 87] The parameter is incorrect
```


### Versions

```Text
Version:             2.7.0
API version:         0.8.3
Python version:      3.10.6
Git commit:          c2833339
Built:               Thu, Dec 1, 2022 4:03 PM
OS/Arch:             win32/AMD64
Profile:             prod
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.39.4
```


### Additional context

_No response_


Discussion:
Can confirm.

In my testing it did not _always_ kill the agent. But it does sometimes.
A very quick Google brought me to https://bugs.python.org/issue42962:
> In Windows, os.kill() is a rather confused function. In particular, it confuses a process ID (pid) with a process group ID (pgid).

It does, however, always fail to cancel the runs, and constantly throws that OSError as it keeps trying to kill the process.

---

In the past I used this in a quick and dirty script to kill a process. I didn't check how this will behave in Prefect's code though.
```py
import signal

def signal_handler(sig, frame):
    logger.info("Exiting")
    kill_process()
    sys.exit()


signal.signal(signal.SIGINT, signal_handler)


def kill_process(pid):
    ...
    subprocess.call(["taskkill", "/pid", str(pid), "/f"])
    ...

```
Thanks a ton yall - just reproduced this and getting on it now! 

Resolving patch:
diff --git a/src/prefect/infrastructure/process.py b/src/prefect/infrastructure/process.py
--- a/src/prefect/infrastructure/process.py
+++ b/src/prefect/infrastructure/process.py
@@ -3,10 +3,11 @@
 import os
 import signal
 import socket
+import subprocess
 import sys
 import tempfile
 from pathlib import Path
-from typing import Tuple, Union
+from typing import Dict, Tuple, Union
 
 import anyio
 import anyio.abc
@@ -19,6 +20,10 @@
 from prefect.utilities.asyncutils import sync_compatible
 from prefect.utilities.processutils import run_process
 
+if sys.platform == "win32":
+    # exit code indicating that the process was terminated by Ctrl+C or Ctrl+Break
+    STATUS_CONTROL_C_EXIT = 0xC000013A
+
 
 def _use_threaded_child_watcher():
     if (
@@ -99,6 +104,13 @@ async def run(
                 f"Process{display_name} running command: {' '.join(self.command)} in {working_dir}"
             )
 
+            # We must add creationflags to a dict so it is only passed as a function
+            # parameter on Windows, because the presence of creationflags causes
+            # errors on Unix even if set to None
+            kwargs: Dict[str, object] = {}
+            if sys.platform == "win32":
+                kwargs["creationflags"] = subprocess.CREATE_NEW_PROCESS_GROUP
+
             process = await run_process(
                 self.command,
                 stream_output=self.stream_output,
@@ -106,6 +118,7 @@ async def run(
                 task_status_handler=_infrastructure_pid_from_process,
                 env=self._get_environment_variables(),
                 cwd=working_dir,
+                **kwargs,
             )
 
         # Use the pid for display if no name was given
@@ -130,6 +143,13 @@ async def run(
                     "This indicates that the process was terminated due to high "
                     "memory usage."
                 )
+            elif (
+                sys.platform == "win32" and process.returncode == STATUS_CONTROL_C_EXIT
+            ):
+                help_message = (
+                    f"Process was terminated due to a Ctrl+C or Ctrl+Break signal. "
+                    "Typically, this is caused by manual cancellation."
+                )
 
             self.logger.error(
                 f"Process{display_name} exited with status code: "
@@ -150,16 +170,17 @@ async def kill(self, infrastructure_pid: str, grace_seconds: int = 30):
                 f"Unable to kill process {pid!r}: The process is running on a different host {hostname!r}."
             )
 
-        # In a non-windows enviornment first send a SIGTERM, then, after
+        # In a non-windows environment first send a SIGTERM, then, after
         # `grace_seconds` seconds have passed subsequent send SIGKILL. In
         # Windows we use CTRL_BREAK_EVENT as SIGTERM is useless:
         # https://bugs.python.org/issue26350
         if sys.platform == "win32":
             try:
                 os.kill(pid, signal.CTRL_BREAK_EVENT)
-            except ProcessLookupError:
-                # The process exited before we were able to kill it.
-                return
+            except (ProcessLookupError, WindowsError):
+                raise InfrastructureNotFound(
+                    f"Unable to kill process {pid!r}: The process was not found."
+                )
         else:
             try:
                 os.kill(pid, signal.SIGTERM)
@@ -185,7 +206,7 @@ async def kill(self, infrastructure_pid: str, grace_seconds: int = 30):
 
             try:
                 os.kill(pid, signal.SIGKILL)
-            except ProcessLookupError:
+            except OSError:
                 # We shouldn't ever end up here, but it's possible that the
                 # process ended right after the check above.
                 return
diff --git a/src/prefect/utilities/processutils.py b/src/prefect/utilities/processutils.py
--- a/src/prefect/utilities/processutils.py
+++ b/src/prefect/utilities/processutils.py
@@ -1,10 +1,23 @@
+import asyncio
 import os
 import signal
 import subprocess
 import sys
 from contextlib import asynccontextmanager
+from dataclasses import dataclass
 from io import TextIOBase
-from typing import Any, Callable, List, Optional, TextIO, Tuple, Union
+from typing import (
+    IO,
+    Any,
+    Callable,
+    List,
+    Mapping,
+    Optional,
+    Sequence,
+    TextIO,
+    Tuple,
+    Union,
+)
 
 import anyio
 import anyio.abc
@@ -13,6 +26,157 @@
 TextSink = Union[anyio.AsyncFile, TextIO, TextSendStream]
 
 
+if sys.platform == "win32":
+    from ctypes import WINFUNCTYPE, c_int, c_uint, windll
+
+    _windows_process_group_pids = set()
+
+    @WINFUNCTYPE(c_int, c_uint)
+    def _win32_ctrl_handler(dwCtrlType):
+        """
+        A callback function for handling CTRL events cleanly on Windows. When called,
+        this function will terminate all running win32 subprocesses the current
+        process started in new process groups.
+        """
+        for pid in _windows_process_group_pids:
+            try:
+                os.kill(pid, signal.CTRL_BREAK_EVENT)
+            except OSError:
+                # process is already terminated
+                pass
+
+        # returning 0 lets the next handler in the chain handle the signal
+        return 0
+
+    # anyio process wrapper classes
+    @dataclass(eq=False)
+    class StreamReaderWrapper(anyio.abc.ByteReceiveStream):
+        _stream: asyncio.StreamReader
+
+        async def receive(self, max_bytes: int = 65536) -> bytes:
+            data = await self._stream.read(max_bytes)
+            if data:
+                return data
+            else:
+                raise anyio.EndOfStream
+
+        async def aclose(self) -> None:
+            self._stream.feed_eof()
+
+    @dataclass(eq=False)
+    class StreamWriterWrapper(anyio.abc.ByteSendStream):
+        _stream: asyncio.StreamWriter
+
+        async def send(self, item: bytes) -> None:
+            self._stream.write(item)
+            await self._stream.drain()
+
+        async def aclose(self) -> None:
+            self._stream.close()
+
+    @dataclass(eq=False)
+    class Process(anyio.abc.Process):
+        _process: asyncio.subprocess.Process
+        _stdin: Union[StreamWriterWrapper, None]
+        _stdout: Union[StreamReaderWrapper, None]
+        _stderr: Union[StreamReaderWrapper, None]
+
+        async def aclose(self) -> None:
+            if self._stdin:
+                await self._stdin.aclose()
+            if self._stdout:
+                await self._stdout.aclose()
+            if self._stderr:
+                await self._stderr.aclose()
+
+            await self.wait()
+
+        async def wait(self) -> int:
+            return await self._process.wait()
+
+        def terminate(self) -> None:
+            self._process.terminate()
+
+        def kill(self) -> None:
+            self._process.kill()
+
+        def send_signal(self, signal: int) -> None:
+            self._process.send_signal(signal)
+
+        @property
+        def pid(self) -> int:
+            return self._process.pid
+
+        @property
+        def returncode(self) -> Union[int, None]:
+            return self._process.returncode
+
+        @property
+        def stdin(self) -> Union[anyio.abc.ByteSendStream, None]:
+            return self._stdin
+
+        @property
+        def stdout(self) -> Union[anyio.abc.ByteReceiveStream, None]:
+            return self._stdout
+
+        @property
+        def stderr(self) -> Union[anyio.abc.ByteReceiveStream, None]:
+            return self._stderr
+
+    async def _open_anyio_process(
+        command: Union[str, bytes, Sequence[Union[str, bytes]]],
+        *,
+        stdin: Union[int, IO[Any], None] = None,
+        stdout: Union[int, IO[Any], None] = None,
+        stderr: Union[int, IO[Any], None] = None,
+        cwd: Union[str, bytes, os.PathLike, None] = None,
+        env: Union[Mapping[str, str], None] = None,
+        start_new_session: bool = False,
+        **kwargs,
+    ):
+        """
+        Open a subprocess and return a `Process` object.
+
+        Args:
+            command: The command to run
+            kwargs: Additional arguments to pass to `asyncio.create_subprocess_exec`
+
+        Returns:
+            A `Process` object
+        """
+        # call either asyncio.create_subprocess_exec or asyncio.create_subprocess_shell
+        # depending on whether the command is a list or a string
+        if isinstance(command, list):
+            process = await asyncio.create_subprocess_exec(
+                *command,
+                stdin=stdin,
+                stdout=stdout,
+                stderr=stderr,
+                cwd=cwd,
+                env=env,
+                start_new_session=start_new_session,
+                **kwargs,
+            )
+        else:
+            process = await asyncio.create_subprocess_shell(
+                command,
+                stdin=stdin,
+                stdout=stdout,
+                stderr=stderr,
+                cwd=cwd,
+                env=env,
+                start_new_session=start_new_session,
+                **kwargs,
+            )
+
+        return Process(
+            process,
+            StreamWriterWrapper(process.stdin) if process.stdin else None,
+            StreamReaderWrapper(process.stdout) if process.stdout else None,
+            StreamReaderWrapper(process.stderr) if process.stderr else None,
+        )
+
+
 @asynccontextmanager
 async def open_process(command: List[str], **kwargs):
     """
@@ -29,10 +193,27 @@ async def open_process(command: List[str], **kwargs):
             "The command passed to open process must be a list. You passed the command"
             f"'{command}', which is type '{type(command)}'."
         )
+
     if sys.platform == "win32":
         command = " ".join(command)
+        process = await _open_anyio_process(command, **kwargs)
+    else:
+        process = await anyio.open_process(command, **kwargs)
 
-    process = await anyio.open_process(command, **kwargs)
+    # if there's a creationflags kwarg and it contains CREATE_NEW_PROCESS_GROUP,
+    # use SetConsoleCtrlHandler to handle CTRL-C
+    win32_process_group = False
+    if (
+        sys.platform == "win32"
+        and "creationflags" in kwargs
+        and kwargs["creationflags"] & subprocess.CREATE_NEW_PROCESS_GROUP
+    ):
+        win32_process_group = True
+        _windows_process_group_pids.add(process.pid)
+        # Add a handler for CTRL-C. Re-adding the handler is safe as Windows
+        # will not add a duplicate handler if _win32_ctrl_handler is
+        # already registered.
+        windll.kernel32.SetConsoleCtrlHandler(_win32_ctrl_handler, 1)
 
     try:
         async with process:
@@ -40,7 +221,10 @@ async def open_process(command: List[str], **kwargs):
     finally:
         try:
             process.terminate()
-        except ProcessLookupError:
+            if win32_process_group:
+                _windows_process_group_pids.remove(process.pid)
+
+        except OSError:
             # Occurs if the process is already terminated
             pass
 


Test patch:
diff --git a/tests/infrastructure/test_process.py b/tests/infrastructure/test_process.py
--- a/tests/infrastructure/test_process.py
+++ b/tests/infrastructure/test_process.py
@@ -1,6 +1,7 @@
 import os
 import signal
 import socket
+import subprocess
 import sys
 from unittest.mock import MagicMock, call
 
@@ -8,6 +9,10 @@
 import anyio.abc
 import pytest
 
+import prefect
+import prefect.infrastructure
+import prefect.infrastructure.process
+import prefect.utilities.processutils
 from prefect.exceptions import InfrastructureNotAvailable, InfrastructureNotFound
 from prefect.infrastructure.process import Process
 from prefect.testing.utilities import AsyncMock
@@ -15,9 +20,20 @@
 
 @pytest.fixture
 def mock_open_process(monkeypatch):
-    monkeypatch.setattr("anyio.open_process", AsyncMock())
-    anyio.open_process.return_value.terminate = MagicMock()  #  Not an async attribute
-    yield anyio.open_process
+    if sys.platform == "win32":
+        monkeypatch.setattr(
+            "prefect.utilities.processutils._open_anyio_process", AsyncMock()
+        )
+        prefect.utilities.processutils._open_anyio_process.return_value.terminate = (  # noqa
+            MagicMock()
+        )
+
+        yield prefect.utilities.processutils._open_anyio_process  # noqa
+    else:
+        monkeypatch.setattr("anyio.open_process", AsyncMock())
+        anyio.open_process.return_value.terminate = MagicMock()  # noqa
+
+        yield anyio.open_process
 
 
 @pytest.mark.parametrize("stream_output", [True, False])
@@ -56,23 +72,53 @@ def test_process_returns_exit_code(exit_code):
 
 
 def test_process_runs_command(tmp_path):
-    # Perform a side-effect to demonstrate the command is run
-    assert Process(command=["touch", str(tmp_path / "canary")]).run()
-    assert (tmp_path / "canary").exists()
+    """
+    Run a command that creates a file as a side effect to demonstrate the command
+    is run.
+    """
+    file_path = tmp_path / "test.txt"
+    # don't assume any specific programs are available other than touch on
+    # Unix-like systems and cmd.exe on Windows
+    if sys.platform == "win32":
+        command = ["cmd.exe", "/c", f"echo hello > {str(file_path)}"]
+    else:
+        command = ["touch", str(file_path)]
+
+    assert Process(command=command).run()
+    assert file_path.exists()
 
 
 def test_process_runs_command_in_working_dir_str(tmpdir, capsys):
-    assert Process(
-        command=["bash", "-c", "pwd"], stream_output=True, working_dir=str(tmpdir)
-    ).run()
+    """
+    Test that a command that displays the current working directory runs
+    and displays the specified working directory for the process when the
+    working directory is a string.
+    """
+
+    # a command to display the current working directory with alternatives
+    # for Windows and Unix-like systems
+    if sys.platform == "win32":
+        command = ["cmd.exe", "/c", "cd"]
+    else:
+        command = ["bash", "-c", "pwd"]
+
+    assert Process(command=command, stream_output=True, working_dir=str(tmpdir)).run()
     out, _ = capsys.readouterr()
     assert str(tmpdir) in out
 
 
 def test_process_runs_command_in_working_dir_path(tmp_path, capsys):
-    assert Process(
-        command=["bash", "-c", "pwd"], stream_output=True, working_dir=tmp_path
-    ).run()
+    """
+    Test that a command that displays the current working directory runs
+    and displays the specified working directory for the process when the
+    working directory is a path.
+    """
+    if sys.platform == "win32":
+        command = ["cmd.exe", "/c", "cd"]
+    else:
+        command = ["bash", "-c", "pwd"]
+
+    assert Process(command=command, stream_output=True, working_dir=tmp_path).run()
     out, _ = capsys.readouterr()
     assert str(tmp_path) in out
 
@@ -207,9 +253,6 @@ async def test_process_kill_mismatching_hostname(monkeypatch):
 
 
 async def test_process_kill_no_matching_pid(monkeypatch):
-    os_kill = MagicMock(side_effect=ProcessLookupError())
-    monkeypatch.setattr("os.kill", os_kill)
-
     infrastructure_pid = f"{socket.gethostname()}:12345"
 
     process = Process(command=["noop"])
@@ -288,3 +331,47 @@ async def test_process_kill_windows_sends_ctrl_break(monkeypatch):
     )
 
     os_kill.assert_called_once_with(12345, signal.CTRL_BREAK_EVENT)
+
+
+@pytest.mark.skipif(
+    sys.platform != "win32",
+    reason="subprocess.CREATE_NEW_PROCESS_GROUP is only defined in Windows",
+)
+def test_windows_process_run_sets_process_group_creation_flag(monkeypatch):
+    mock_process = AsyncMock()
+    mock_process.returncode = 0
+    mock_process.terminate = AsyncMock()
+
+    mock_run_process_call = AsyncMock(return_value=mock_process)
+
+    monkeypatch.setattr(
+        prefect.infrastructure.process, "run_process", mock_run_process_call
+    )
+
+    prefect.infrastructure.Process(command=["echo", "hello world"]).run()
+
+    mock_run_process_call.assert_awaited_once()
+    (_, kwargs) = mock_run_process_call.call_args
+    assert kwargs.get("creationflags") == subprocess.CREATE_NEW_PROCESS_GROUP
+
+
+@pytest.mark.skipif(
+    sys.platform == "win32",
+    reason="The asyncio.open_process_*.creationflags argument is only supported on Windows",
+)
+def test_unix_process_run_does_not_set_creation_flag(monkeypatch):
+    mock_process = AsyncMock()
+    mock_process.returncode = 0
+    mock_process.terminate = AsyncMock()
+
+    mock_run_process_call = AsyncMock(return_value=mock_process)
+
+    monkeypatch.setattr(
+        prefect.infrastructure.process, "run_process", mock_run_process_call
+    )
+
+    prefect.infrastructure.Process(command=["echo", "hello world"]).run()
+
+    mock_run_process_call.assert_awaited_once()
+    (_, kwargs) = mock_run_process_call.call_args
+    assert kwargs.get("creationflags") is None
diff --git a/tests/utilities/test_processutils.py b/tests/utilities/test_processutils.py
--- a/tests/utilities/test_processutils.py
+++ b/tests/utilities/test_processutils.py
@@ -1,9 +1,11 @@
 import signal
+import subprocess
 import sys
 from unittest import mock
 
 import pytest
 
+import prefect.utilities.processutils
 from prefect.utilities.processutils import kill_on_interrupt, open_process, run_process
 
 
@@ -68,6 +70,45 @@ async def test_runs_if_cmd_is_list(self):
         async with open_process(self.list_cmd) as process:
             assert process
 
+    @pytest.mark.skipif(
+        sys.platform != "win32",
+        reason="CTRL_C_HANDLER is only defined in Windows",
+    )
+    async def test_adds_ctrl_c_handler_to_win32_process_group(self, monkeypatch):
+        """
+        If the process is a Windows process group, we need to add a handler for
+        CTRL_C_EVENT to the process group so we can kill the process group
+        when the user presses CTRL+C.
+        """
+        mock_ctrl_c_handler = mock.Mock()
+        monkeypatch.setattr(
+            prefect.utilities.processutils, "_win32_ctrl_handler", mock_ctrl_c_handler
+        )
+        mock_set_console_ctrl_handler = mock.Mock()
+        monkeypatch.setattr(
+            prefect.utilities.processutils.windll.kernel32,
+            "SetConsoleCtrlHandler",
+            mock_set_console_ctrl_handler,
+        )
+
+        mock_process = mock.AsyncMock()
+        mock_process.terminate = mock.MagicMock()
+        mock_open_process = mock.AsyncMock(return_value=mock_process)
+        monkeypatch.setattr(
+            prefect.utilities.processutils, "_open_anyio_process", mock_open_process
+        )
+
+        await prefect.utilities.processutils.run_process(
+            self.list_cmd, creationflags=subprocess.CREATE_NEW_PROCESS_GROUP
+        )
+        mock_open_process.assert_called_once_with(
+            " ".join(self.list_cmd),
+            stdout=mock.ANY,
+            stderr=mock.ANY,
+            creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,
+        )
+        mock_set_console_ctrl_handler.assert_called_once_with(mock_ctrl_c_handler, 1)
+
 
 class TestKillOnInterrupt:
     @pytest.mark.skipif(