Repository: PrefectHQ/prefect

Issue:
Use literal_eval for parsing configuration values
## Current behavior
*Please describe how the feature works today*
Currently, Prefect takes a [hardcoded stab](https://github.com/PrefectHQ/prefect/blob/master/src/prefect/configuration.py#L37) at converting strings to Python objects. We could replace this function with `ast.literal_eval()` and instantly gain a more flexible cast, including the ability to natively handle lists and dicts. 



## Proposed behavior
*Please describe your proposed change to the current behavior*

Currently, setting `PREFECT__X__Y="['a', 'b', 3]"` results in `prefect.config.x.y == "['a', 'b', 3]"`. This means some Prefect objects (thinking of environments, but also including the config itself) must re-parse config options. This is after we already make an attempt to coerce some values to known types. Replacing this with `ast.literal_eval()` would result in `prefect.config.x.y == ['a', 'b', 3]`.


## Example
*Please give an example of how the enhancement would be useful*



Discussion:
I definitely get the idea here @jlowin but I think using `json` might introduce more user-facing complexity; to give you a concrete example, the code you wrote above would _not_ work using `json.loads` because `json` expects double quotes.  

That being said, using `ast.literal_eval` avoids those sorts of formatting issues and still allows for the safety + flexibility that I think you're looking for here.
Ah shoot you're right, that's not even valid json. I like your suggestion better! 

(Updated the description to reflect literal_eval())

Resolving patch:
diff --git a/src/prefect/configuration.py b/src/prefect/configuration.py
--- a/src/prefect/configuration.py
+++ b/src/prefect/configuration.py
@@ -1,6 +1,7 @@
 import datetime
 import os
 import re
+from ast import literal_eval
 from typing import Optional, Union, cast
 
 import toml
@@ -41,14 +42,13 @@ def string_to_type(val: str) -> Union[bool, int, float, str]:
     Maps:
         - "true" (any capitalization) to `True`
         - "false" (any capitalization) to `False`
-        - integers to `int`
-        - floats to `float`
-
+        - any other valid literal Python syntax interpretable by ast.literal_eval
+        
     Arguments:
         - val (str): the string value of an environment variable
 
     Returns:
-        Union[bool, int, float, str]: the type-cast env var value
+        Union[bool, int, float, str, dict, list, None, tuple]: the type-cast env var value
     """
 
     # bool
@@ -57,19 +57,10 @@ def string_to_type(val: str) -> Union[bool, int, float, str]:
     elif val.upper() == "FALSE":
         return False
 
-    # int
-    try:
-        val_as_int = int(val)
-        if str(val_as_int) == val:
-            return val_as_int
-    except Exception:
-        pass
-
-    # float
+    # dicts, ints, floats, or any other literal Python syntax
     try:
-        val_as_float = float(val)
-        if str(val_as_float) == val:
-            return val_as_float
+        val_as_obj = literal_eval(val)
+        return val_as_obj
     except Exception:
         pass
 


Test patch:
diff --git a/tests/test_configuration.py b/tests/test_configuration.py
--- a/tests/test_configuration.py
+++ b/tests/test_configuration.py
@@ -64,6 +64,10 @@ def config(test_config_file_path, monkeypatch):
     monkeypatch.setenv("PREFECT_TEST__ENV_VARS__NEGATIVE_INT", "-10")
     monkeypatch.setenv("PREFECT_TEST__ENV_VARS__FLOAT", "7.5")
     monkeypatch.setenv("PREFECT_TEST__ENV_VARS__NEGATIVE_FLOAT", "-7.5")
+    monkeypatch.setenv(
+        "PREFECT_TEST__CONTEXT__SECRETS__AWS_CREDENTIALS",
+        '{"ACCESS_KEY": "abcdef", "SECRET_ACCESS_KEY": "ghijklmn"}',
+    )
     monkeypatch.setenv("PATH", "1/2/3")
     monkeypatch.setenv(
         "PREFECT_TEST__ENV_VARS__ESCAPED_CHARACTERS", r"line 1\nline 2\rand 3\tand 4"
@@ -81,6 +85,12 @@ def test_keys(config):
     assert "x" not in config
 
 
+def test_dicts_are_created(config):
+    val = config.context.secrets.AWS_CREDENTIALS
+    assert isinstance(val, dict)
+    assert val["ACCESS_KEY"] == "abcdef"
+
+
 def test_getattr_missing(config):
     with pytest.raises(AttributeError, match="object has no attribute"):
         config.hello