Repository: PrefectHQ/prefect

Issue:
PREFECT_TASK_DEFAULT_RETRY_DELAY_SECONDS environment variable does not accept a list
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

When passing a list to the environment variable PREFECT_TASK_DEFAULT_RETRY_DELAY_SECONDS the server is unable to start up.

### Reproduction

```python3
PREFECT_TASK_DEFAULT_RETRY_DELAY_SECONDS=[10, 20, 40]
```


### Error

```python3
Error loading ASGI app factory: unhashable type: 'list'
Server stopped!
```


### Versions

```Text
Version:             2.14.13
API version:         0.8.4
Python version:      3.11.7
Git commit:          9df5ba72
Built:               Thu, Jan 4, 2024 5:09 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Server:
  Database:          postgresql
```


### Additional context

https://github.com/PrefectHQ/prefect/blob/cbd8b8f0f80b505077da927094eb6f9b44f6ba30/src/prefect/settings.py#L755-L757


Discussion:
Thanks for the bug report @beornf. I was able to reproduce the issue running `prefect server start` - it looks like we're caching [apps based on discovered server settings ](https://github.com/PrefectHQ/prefect/blob/3ca9bc01583137ba2360f1c982fe481423176b22/src/prefect/server/api/server.py#L425) and we'll want to make sure this isn't a problem with any other server settings that could be parsed as more complex, unhashable types.

Can I ask why you're trying to set `PREFECT_TASK_DEFAULT_RETRY_DELAY_SECONDS` for `prefect server start`? It looks like that specific setting doesn't configure server behavior, so it doesn't need to be set at all.
The environment variable `PREFECT_TASK_DEFAULT_RETRY_DELAY_SECONDS` sets the default task retry delay the same as `prefect config set` right? I just want to set global defaults when starting up a prefect docker container.

Resolving patch:
diff --git a/src/prefect/server/api/server.py b/src/prefect/server/api/server.py
--- a/src/prefect/server/api/server.py
+++ b/src/prefect/server/api/server.py
@@ -501,7 +501,7 @@ def create_app(
             and services will be disabled.
     """
     settings = settings or prefect.settings.get_current_settings()
-    cache_key = (settings, ephemeral)
+    cache_key = (settings.hash_key(), ephemeral)
 
     if cache_key in APP_CACHE and not ignore_cache:
         return APP_CACHE[cache_key]
diff --git a/src/prefect/settings.py b/src/prefect/settings.py
--- a/src/prefect/settings.py
+++ b/src/prefect/settings.py
@@ -1660,6 +1660,14 @@ def with_obfuscated_secrets(self):
         settings.__fields_set__.intersection_update(self.__fields_set__)
         return settings
 
+    def hash_key(self) -> str:
+        """
+        Return a hash key for the settings object.  This is needed since some
+        settings may be unhashable.  An example is lists.
+        """
+        env_variables = self.to_environment_variables()
+        return str(hash(tuple((key, value) for key, value in env_variables.items())))
+
     def to_environment_variables(
         self, include: Iterable[Setting] = None, exclude_unset: bool = False
     ) -> Dict[str, str]:


Test patch:
diff --git a/tests/test_engine.py b/tests/test_engine.py
--- a/tests/test_engine.py
+++ b/tests/test_engine.py
@@ -1728,7 +1728,7 @@ async def test_global_task_retry_delay_seconds(
         self, mock_anyio_sleep, prefect_client, flow_run, result_factory
     ):
         with temporary_settings(
-            updates={PREFECT_TASK_DEFAULT_RETRY_DELAY_SECONDS: "43"}
+            updates={PREFECT_TASK_DEFAULT_RETRY_DELAY_SECONDS: [3, 5, 9]}
         ):
             # the flow run must be running prior to running tasks
             await prefect_client.set_flow_run_state(
@@ -1739,14 +1739,14 @@ async def test_global_task_retry_delay_seconds(
             # Define a task that fails once and then succeeds
             mock = MagicMock()
 
-            @task(retries=1)
+            @task(retries=3)
             def flaky_function():
                 mock()
 
-                if mock.call_count == 2:
+                if mock.call_count == 4:
                     return 1
 
-                raise ValueError("try again, but only once")
+                raise ValueError("try again")
 
             # Create a task run to test
             task_run = await prefect_client.create_task_run(
@@ -1757,7 +1757,7 @@ def flaky_function():
             )
 
             # Actually run the task
-            with mock_anyio_sleep.assert_sleeps_for(43):
+            with mock_anyio_sleep.assert_sleeps_for(17):
                 await orchestrate_task_run(
                     task=flaky_function,
                     task_run=task_run,
@@ -1769,6 +1769,8 @@ def flaky_function():
                     log_prints=False,
                 )
 
+            assert mock_anyio_sleep.await_count == 3
+
     async def test_retry_condition_fn_retries_after_failure(
         self, mock_anyio_sleep, prefect_client, flow_run, result_factory
     ):
diff --git a/tests/test_settings.py b/tests/test_settings.py
--- a/tests/test_settings.py
+++ b/tests/test_settings.py
@@ -243,6 +243,14 @@ def test_settings_to_environment_does_not_use_value_callback(self):
         # generating the environment variables
         assert "PREFECT_UI_API_URL" not in settings.to_environment_variables()
 
+    def test_settings_hash_key(self):
+        settings = Settings(PREFECT_TEST_MODE=True)
+        diff_settings = Settings(PREFECT_TEST_MODE=False)
+
+        assert settings.hash_key() == settings.hash_key()
+
+        assert settings.hash_key() != diff_settings.hash_key()
+
     @pytest.mark.parametrize(
         "log_level_setting",
         [