Repository: PrefectHQ/prefect

Issue:
The `prefect dev start` command fails with an error
### First check

- [X] I am a contributor to the Prefect codebase

### Description

When I run `prefect dev start`, a Python stack trace prints on the screen, and the Orion API server fails to start. The CLI command process is still active because I need to press Control-C to terminate it.

```
env ❯ prefect dev start
Running: uvicorn --factory prefect.orion.api.server:create_app --host 127.0.0.1 --port 4200 --log-level debug --reload --reload-dir
/Users/andrew/src/prefect/src/prefect
Installing npm packages...
Starting UI development server...
Creating hot-reloading agent process...
INFO:     Will watch for changes in these directories: ['/Users/andrew/src/prefect/src/prefect']
INFO:     Uvicorn running on http://127.0.0.1:4200/ (Press CTRL+C to quit)
INFO:     Started reloader process [23264] using WatchFiles

> ui-orion@0.1.0 serve
> vite --host


  vite v2.9.15 dev server running at:

  > Local:    http://localhost:3000/
  > Network:  http://192.168.68.116:3000/
  > Network:  http://100.87.38.18:3000/
  > Network:  http://192.168.0.195:3000/

  ready in 348ms.

Traceback (most recent call last):
  File "/Users/andrew/src/prefect/src/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/andrew/src/prefect/src/prefect/utilities/asyncutils.py", line 205, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/Users/andrew/src/prefect/src/prefect/utilities/asyncutils.py", line 156, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/Users/andrew/src/prefect/env/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/andrew/src/prefect/env/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/andrew/src/prefect/env/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/andrew/src/prefect/src/prefect/cli/agent.py", line 92, in start
    q = await client.read_work_queue(UUID(work_queue))
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/uuid.py", line 174, in __init__
    hex = hex.replace('urn:', '').replace('uuid:', '')
AttributeError: 'ArgumentInfo' object has no attribute 'replace'
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/Users/andrew/src/prefect/src/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/andrew/src/prefect/src/prefect/utilities/asyncutils.py", line 205, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/Users/andrew/src/prefect/src/prefect/utilities/asyncutils.py", line 156, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/Users/andrew/src/prefect/env/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/andrew/src/prefect/env/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/andrew/src/prefect/env/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/andrew/src/prefect/src/prefect/cli/agent.py", line 92, in start
    q = await client.read_work_queue(UUID(work_queue))
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/uuid.py", line 174, in __init__
    hex = hex.replace('urn:', '').replace('uuid:', '')
AttributeError: 'ArgumentInfo' object has no attribute 'replace'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/andrew/src/prefect/src/prefect/cli/_utilities.py", line 50, in wrapper
    exit_with_error("An exception occurred.")
  File "/Users/andrew/src/prefect/src/prefect/cli/_utilities.py", line 22, in exit_with_error
    app.console.print(message, **kwargs)
AttributeError: 'PrefectTyper' object has no attribute 'console'
```

### Impact

I can't run the API server and UI together for development with a single CLI command. The commands `prefect orion start` and `prefect dev ui` work without error.

### Additional context

```
env ❯ prefect --version
2.7.3+4.gb7e5747d4

env ❯ python --version
Python 3.9.12 
```


Discussion:

Resolving patch:
diff --git a/src/prefect/cli/dev.py b/src/prefect/cli/dev.py
--- a/src/prefect/cli/dev.py
+++ b/src/prefect/cli/dev.py
@@ -11,6 +11,7 @@
 import time
 from functools import partial
 from string import Template
+from typing import List
 
 import anyio
 import typer
@@ -24,6 +25,8 @@
 from prefect.orion.api.server import create_app
 from prefect.settings import (
     PREFECT_API_URL,
+    PREFECT_CLI_COLORS,
+    PREFECT_CLI_WRAP_LINES,
     PREFECT_ORION_API_HOST,
     PREFECT_ORION_API_PORT,
 )
@@ -33,7 +36,8 @@
 DEV_HELP = """
 Commands for development.
 
-Note that many of these commands require extra dependencies (such as npm and MkDocs) to function properly.
+Note that many of these commands require extra dependencies (such as npm and MkDocs)
+to function properly.
 """
 dev_app = PrefectTyper(
     name="dev", short_help="Commands for development.", help=DEV_HELP
@@ -47,13 +51,67 @@ def exit_with_error_if_not_editable_install():
         or not (prefect.__root_path__ / "setup.py").exists()
     ):
         exit_with_error(
-            "Development commands cannot be used without an editable installation of Prefect. "
-            "Development commands require content outside of the 'prefect' module which "
-            "is not available when installed into your site-packages. "
+            "Development commands require an editable Prefect installation. "
+            "Development commands require content outside of the 'prefect' module  "
+            "which is not available when installed into your site-packages. "
             f"Detected module path: {prefect.__module_path__}."
         )
 
 
+def agent_process_entrypoint(**kwargs):
+    """
+    An entrypoint for starting an agent in a subprocess. Adds a Rich console
+    to the Typer app, processes Typer default parameters, then starts an agent.
+    All kwargs are forwarded to  `prefect.cli.agent.start`.
+    """
+    import inspect
+
+    import rich
+
+    # import locally so only the `dev` command breaks if Typer internals change
+    from typer.models import ParameterInfo
+
+    # Typer does not process default parameters when calling a function
+    # directly, so we must set `start_agent`'s default parameters manually.
+    # get the signature of the `start_agent` function
+    start_agent_signature = inspect.signature(start_agent)
+
+    # for any arguments not present in kwargs, use the default value.
+    for name, param in start_agent_signature.parameters.items():
+        if name not in kwargs:
+            # All `param.default` values for start_agent are Typer params that store the
+            # actual default value in their `default` attribute and we must call
+            # `param.default.default` to get the actual default value. We should also
+            # ensure we extract the right default if non-Typer defaults are added
+            # to `start_agent` in the future.
+            if isinstance(param.default, ParameterInfo):
+                default = param.default.default
+            else:
+                default = param.default
+
+            # Some defaults are Prefect `SettingsOption.value` methods
+            # that must be called to get the actual value.
+            kwargs[name] = default() if callable(default) else default
+
+    # add a console, because calling the agent start function directly
+    # instead of via CLI call means `app` has no `console` attached.
+    app.console = (
+        rich.console.Console(
+            highlight=False,
+            color_system="auto" if PREFECT_CLI_COLORS else None,
+            soft_wrap=not PREFECT_CLI_WRAP_LINES.value(),
+        )
+        if not getattr(app, "console", None)
+        else app.console
+    )
+
+    try:
+        start_agent(**kwargs)  # type: ignore
+    except KeyboardInterrupt:
+        # expected when watchfiles kills the process
+        pass
+
+
 @dev_app.command()
 def build_docs(
     schema_path: str = None,
@@ -95,7 +153,7 @@ def build_ui():
                 subprocess.check_output(
                     ["npm", "ci", "install"], shell=sys.platform == "win32"
                 )
-            except Exception as exc:
+            except Exception:
                 app.console.print(
                     "npm call failed - try running `nvm use` first.", style="red"
                 )
@@ -143,9 +201,12 @@ async def api(
     """
     Starts a hot-reloading development API.
     """
+    import watchfiles
+
     server_env = os.environ.copy()
     server_env["PREFECT_ORION_SERVICES_RUN_IN_APP"] = str(services)
     server_env["PREFECT_ORION_SERVICES_UI"] = "False"
+    server_env["PREFECT_ORION_UI_API_URL"] = f"http://{host}:{port}/api"
 
     command = [
         "uvicorn",
@@ -157,18 +218,54 @@ async def api(
         str(port),
         "--log-level",
         log_level.lower(),
-        "--reload",
-        "--reload-dir",
-        str(prefect.__module_path__),
     ]
 
     app.console.print(f"Running: {' '.join(command)}")
+    import signal
 
-    await run_process(command=command, env=server_env, stream_output=True)
+    stop_event = anyio.Event()
+    start_command = partial(
+        run_process, command=command, env=server_env, stream_output=True
+    )
+
+    async with anyio.create_task_group() as tg:
+        try:
+            server_pid = await tg.start(start_command)
+            async for _ in watchfiles.awatch(
+                prefect.__module_path__, stop_event=stop_event  # type: ignore
+            ):
+                # when any watched files change, restart the server
+                app.console.print("Restarting Prefect Server...")
+                os.kill(server_pid, signal.SIGTERM)  # type: ignore
+                # start a new server
+                server_pid = await tg.start(start_command)
+        except RuntimeError as err:
+            # a bug in watchfiles causes an 'Already borrowed' error from Rust when
+            # exiting: https://github.com/samuelcolvin/watchfiles/issues/200
+            if str(err).strip() != "Already borrowed":
+                raise
+        except KeyboardInterrupt:
+            # exit cleanly on ctrl-c by killing the server process if it's
+            # still running
+            try:
+                os.kill(server_pid, signal.SIGTERM)  # type: ignore
+            except ProcessLookupError:
+                # process already exited
+                pass
+
+            stop_event.set()
 
 
 @dev_app.command()
-async def agent(api_url: str = SettingsOption(PREFECT_API_URL)):
+async def agent(
+    api_url: str = SettingsOption(PREFECT_API_URL),
+    work_queues: List[str] = typer.Option(
+        ["default"],
+        "-q",
+        "--work-queue",
+        help="One or more work queue names for the agent to pull from.",
+    ),
+):
     """
     Starts a hot-reloading development agent process.
     """
@@ -176,11 +273,18 @@ async def agent(api_url: str = SettingsOption(PREFECT_API_URL)):
     import watchfiles
 
     app.console.print("Creating hot-reloading agent process...")
-    await watchfiles.arun_process(
-        prefect.__module_path__,
-        target=start_agent,
-        kwargs=dict(hide_welcome=False, api=api_url),
-    )
+
+    try:
+        await watchfiles.arun_process(
+            prefect.__module_path__,
+            target=agent_process_entrypoint,
+            kwargs=dict(api=api_url, work_queues=work_queues),
+        )
+    except RuntimeError as err:
+        # a bug in watchfiles causes an 'Already borrowed' error from Rust when
+        # exiting: https://github.com/samuelcolvin/watchfiles/issues/200
+        if str(err).strip() != "Already borrowed":
+            raise
 
 
 @dev_app.command()
@@ -188,6 +292,12 @@ async def start(
     exclude_api: bool = typer.Option(False, "--no-api"),
     exclude_ui: bool = typer.Option(False, "--no-ui"),
     exclude_agent: bool = typer.Option(False, "--no-agent"),
+    work_queues: List[str] = typer.Option(
+        ["default"],
+        "-q",
+        "--work-queue",
+        help="One or more work queue names for the dev agent to pull from.",
+    ),
 ):
     """
     Starts a hot-reloading development server with API, UI, and agent processes.
@@ -209,10 +319,10 @@ async def start(
         if not exclude_agent:
             # Hook the agent to the hosted API if running
             if not exclude_api:
-                host = f"http://{PREFECT_ORION_API_HOST.value()}:{PREFECT_ORION_API_PORT.value()}/api"
+                host = f"http://{PREFECT_ORION_API_HOST.value()}:{PREFECT_ORION_API_PORT.value()}/api"  # noqa
             else:
                 host = PREFECT_API_URL.value()
-            tg.start_soon(agent, host)
+            tg.start_soon(agent, host, work_queues)
 
 
 @dev_app.command()
@@ -311,7 +421,7 @@ def container(bg: bool = False, name="prefect-dev", api: bool = True):
         command=[
             "/bin/bash",
             "-c",
-            f"pip install -e /opt/prefect/repo\\[dev\\] && touch /READY && {blocking_cmd}",
+            f"pip install -e /opt/prefect/repo\\[dev\\] && touch /READY && {blocking_cmd}",  # noqa
         ],
         name=name,
         auto_remove=True,
@@ -332,7 +442,7 @@ def container(bg: bool = False, name="prefect-dev", api: bool = True):
             ready = result.exit_code == 0
             if not ready:
                 time.sleep(3)
-    except:
+    except BaseException:
         print("\nInterrupted. Stopping container...")
         container.stop()
         raise


Test patch:
diff --git a/tests/cli/test_dev.py b/tests/cli/test_dev.py
new file mode 100644
--- /dev/null
+++ b/tests/cli/test_dev.py
@@ -0,0 +1,167 @@
+import inspect
+
+import watchfiles
+from typer import Option
+
+import prefect
+from prefect.cli.dev import agent_process_entrypoint, start_agent
+from prefect.testing.cli import invoke_and_assert
+from prefect.testing.utilities import AsyncMock, MagicMock
+
+
+def test_dev_start_runs_all_services(monkeypatch):
+    """
+    Test that `prefect dev start` runs all services. This test mocks out the
+    `run_process` function along with the `watchfiles.arun_process` function
+    so the test doesn't actually start any processes; instead, it verifies that
+    the command attempts to start all services correctly.
+    """
+
+    # Mock run_process for UI and API start
+    mock_run_process = AsyncMock()
+
+    # Call task_status.started() if run_process was triggered by an
+    # anyio task group `start` call.
+    def mock_run_process_call(*args, **kwargs):
+        if "task_status" in kwargs:
+            kwargs["task_status"].started()
+
+    mock_run_process.side_effect = mock_run_process_call
+    monkeypatch.setattr(prefect.cli.dev, "run_process", mock_run_process)
+
+    # Mock watchfiles.arun_process for agent start
+    mock_arun_process = AsyncMock()
+    monkeypatch.setattr(watchfiles, "arun_process", mock_arun_process)
+
+    # mock `os.kill` since we're not actually running the processes
+    mock_kill = MagicMock()
+    monkeypatch.setattr(prefect.cli.dev.os, "kill", mock_kill)
+
+    # mock watchfiles.awatch
+    mock_awatch = MagicMock()
+
+    # mock_awatch needs to return an async generator
+    async def async_generator():
+        yield None
+
+    mock_awatch.return_value = async_generator()
+    monkeypatch.setattr(watchfiles, "awatch", mock_awatch)
+
+    invoke_and_assert(["dev", "start"], expected_code=0)
+
+    # check that arun_process was called for the agent
+    mock_arun_process.assert_called_once()
+
+    # ensure one of the calls to run_process was for the UI server
+    mock_run_process.assert_any_call(
+        command=["npm", "run", "serve"], stream_output=True
+    )
+
+    # ensure run_process was called for the API server by checking that
+    # the 'command' passed to one of the calls was an array with "uvicorn"
+    # as the first element
+    uvicorn_called = False
+    for call in mock_run_process.call_args_list:
+        if "command" in call.kwargs and call.kwargs["command"][0] == "uvicorn":
+            uvicorn_called = True
+            break
+
+    assert uvicorn_called
+
+
+def test_agent_subprocess_entrypoint_runs_agent_with_valid_params(monkeypatch):
+    mock_agent_start = MagicMock()
+    monkeypatch.setattr(prefect.cli.dev, "start_agent", mock_agent_start)
+
+    api = "http://127.0.0.1:4200"
+    work_queues = ["default"]
+
+    start_agent_signature = inspect.signature(start_agent)
+    start_agent_params = start_agent_signature.parameters
+    mock_agent_start.__signature__ = start_agent_signature
+
+    agent_process_entrypoint(api=api, work_queues=work_queues)
+
+    # Get the call's kwargs from the mock
+    call_args = mock_agent_start.call_args[1]
+
+    # Ensure that types of the call_args match the types of the start_agent.
+    # This verifies that defaults are extracted from Typer.Argument and
+    # Typer.Option before calling start_agent.
+    for param_name, param in start_agent_params.items():
+        # arguments using Typer.Argument or Typer.Option are implicitly a union of the
+        # declared argument type and NoneType. So, we need to check if the call arg
+        # is either the declared type or NoneType.
+
+        # check if the type is subscripted, because we can't use isinstance
+        # on subscripted types
+        if hasattr(param.annotation, "__origin__"):
+            # if so, use subscripted type
+            arg_type = param.annotation.__origin__
+        else:
+            # otherwise, use the type
+            arg_type = param.annotation
+
+        assert (
+            isinstance(call_args[param_name], arg_type) or call_args[param_name] is None
+        )
+
+    # Ensure that the call args we passed to agent_process_entrypoint are
+    # passed to start_agent and not accidentally overwritten by default values.
+    assert call_args["api"] == api
+    assert call_args["work_queues"] == work_queues
+
+
+# ensures that the default values for the agent start function are extracted
+# correctly in cases where some of them are Typer.OptionInfo or Typer.ArgumentInfo
+# and others are normal Python defaults
+def test_mixed_parameter_default_types(monkeypatch):
+    mock_agent_start = MagicMock()
+    monkeypatch.setattr(prefect.cli.dev, "start_agent", mock_agent_start)
+
+    # mock the signature of start_agent to have a mix of Typer.OptionInfo
+    # and normal Python defaults
+    start_agent_signature = inspect.Signature(
+        parameters=[
+            inspect.Parameter(  # Typer.OptionInfo
+                name="api",
+                kind=inspect.Parameter.KEYWORD_ONLY,
+                default=Option(
+                    None,
+                    "--api",
+                    help="The URL of the Prefect API server",
+                ),
+            ),
+            inspect.Parameter(  # normal Python default
+                name="work_queues",
+                kind=inspect.Parameter.KEYWORD_ONLY,
+                default=["default"],
+            ),
+        ]
+    )
+    mock_agent_start.__signature__ = start_agent_signature
+
+    agent_process_entrypoint()
+
+    # if we get this far without errors, it means the entrypoint function
+    # successfully extracted the mixed default arguments
+    mock_agent_start.assert_called_once()
+
+
+def test_agent_subprocess_entrypoint_adds_typer_console(monkeypatch):
+    """
+    Ensures a Rich console is added to the PrefectTyper's global `app` instance.
+    """
+    start_agent_signature = inspect.signature(start_agent)
+
+    mock_agent_start = MagicMock()
+    mock_agent_start.__signature__ = start_agent_signature
+    monkeypatch.setattr(prefect.cli.dev, "start_agent", mock_agent_start)
+
+    mock_app = MagicMock()
+    monkeypatch.setattr(prefect.cli.dev, "app", mock_app)
+
+    agent_process_entrypoint()
+
+    # ensure the console was added to the app
+    assert mock_app.console is not None