Repository: pydantic/pydantic

Issue:
Settings: Custom parsing of environment variables (non-JSON)
# Settings: Custom parsing of environment variables (non-JSON)

Often we want complex environment variables that are not represented as JSON. One example is a list of items. It's not uncommon to comma-delimit lists like this in bash:

```python
import os
from typing import List

from pydantic import BaseSettings

os.environ['options'] = "a,b,c"

class Settings(BaseSettings):
    options: List

s = Settings()
```

This results in a `JSONDecodeError`. Writing a list of items as valid json is error prone and not human friendly to read in the context of environment variables.

### Workaround

One workaround is to store the variable as valid json, which is tricky to type correctly in lots of systems where you have to enter environment variables:

```
OPTIONS='["a","b","c"]'
```

Another (simplified) workaround is to set `json_loads`. but its not very elegant since `json_loads` doesn't know what field its is parsing, which could be error prone:

```python
import json
import os
from typing import List

from pydantic import BaseSettings, BaseModel, root_validator

os.environ['options'] = "a,b,c"
    
def list_parse_fallback(v):
    try:
        return json.loads(v)
    except Exception as e:
        return v.split(",")

class Settings(BaseSettings):
    options: List
        
    class Config:
        json_loads = list_parse_fallback
       
s = Settings()
```

I can see a couple options for implementing the fix:

### 1. Store the parsing method in the field info extra:

```python
parse_func = lambda x: x.split(",")

class Settings(BaseSettings):
    options: List = Field(..., env_parse=parse_func)
```

If we take this approach, I think that we can update this logic branch:
https://github.com/samuelcolvin/pydantic/blob/42395056e18dfaa3ef299373374ab3b12bb196ac/pydantic/env_settings.py#L170-L174

Adding something like the following:

```python
if field.is_complex():
    if field.extra.get("env_parse", None) is not None:
        try:
            env_val = field.extra["env_parse"](env_val)  # type: ignore
        except ValueError as e:
            raise SettingsError(f'error with custom parsing function for "{env_name}"') from e
    else:
        try:
            env_val = self.__config__.json_loads(env_val)  # type: ignore
        except ValueError as e:
            raise SettingsError(f'error parsing JSON for "{env_name}"') from e
d[field.alias] = env_val
```

### 2. Add a new config option just for Settings for overriding how env vars are parsed
Another implementation option is to add a new property like `Settings.Config.parse_env_var` which takes the `field` and the value so that it can be overridden to handle dispatching to different parsing methods for different names/properties of `field` (currently, just overriding `json_loads` means you are passed a value without knowing where it will be stored so you have to test for and handle all of the possible settings values.

```python
class BaseSettings:
    ...
    class Config:
         ...
        @classmethod
        def parse_env_var(cls, field, raw_val):
            return cls.json_loads(raw_val)
```

Then the following line is the only change that is needed:
https://github.com/samuelcolvin/pydantic/blob/master/pydantic/env_settings.py#L62

Changes to `self.__config__.parse_env_var(field, env_val)`


### 3. Call field validators on the raw string instead of trying to load from json first

Change the same line to:
```
env_val, ee = field.validate(env_val)

# collect ee and raise errors
```

Pros:
 - Adding validators for fields is well documented / understood

Cons:
 - Breaks existing JSON functionality if those fields already have validators
 - Mixes up the abstractions in that the functions would now do parsing and validation

### 4. Custom (de)serialization

Let fields implement custom serialization/deserialization methods. Currently there is `json_encoders` but not an equivalent `json_decoders` for use per-field.

There's some discussion of this here: https://github.com/samuelcolvin/pydantic/issues/951

### 5. Something else

Other ideas? Happy to implement a different suggestion.

--------------

Output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())"`:
```
             pydantic version: 1.5.1
            pydantic compiled: True
                 install path: /Users/bull/miniconda3/envs/sandbox/lib/python3.7/site-packages/pydantic
               python version: 3.7.6 (default, Jan  8 2020, 13:42:34)  [Clang 4.0.1 (tags/RELEASE_401/final)]
                     platform: Darwin-19.4.0-x86_64-i386-64bit
     optional deps. installed: []

```




Discussion:
I think we should emove the `if field.is_complex()` bit of `BaseSettings` and replace it with a universal validator `validate('*', pre=True)` which tries decoding JSON, but also in the case of lists supports comma separated lists.

That way you could easily override the validator if you so wished.

I think this should be backwards compatible so could be done before v2. PR welcome.
Hi @samuelcolvin, I took a stab at your suggested implementation of moving the json decoding out of the `_build_environ` method and into a universal validator. Here is what that looks like (before trying to add in any parsing of comma-separated lists):

```python
class BaseSettings(BaseModel):

    ...

    @validator('*', pre=True)
    def validate_env_vars(cls, v, config, field):
        if isinstance(v, str) and field.is_complex():
            try:
                return config.json_loads(v)  # type: ignore
            except ValueError as e:
                raise SettingsError(f'error parsing JSON for "{field}"') from e
        return v
```

Unfortunately, this implementation ran into a problem with two tests: [`test_nested_env_with_basemodel`](https://github.com/samuelcolvin/pydantic/blob/5067508eca6222b9315b1e38a30ddc975dee05e7/tests/test_settings.py#L56-L68) and [`test_nested_env_with_dict`](https://github.com/samuelcolvin/pydantic/blob/5067508eca6222b9315b1e38a30ddc975dee05e7/tests/test_settings.py#L71-L79). 

These two tests have an field `top` that is an inner mapping with fields `apple` and `banana`. These two fields are specified in different places and stitched together: dict `{'apple': 'value'}` is passed into the constructor and the string `'{"banana": "secret_value"}'` is set as an environment variable. In the [previous implementation](https://github.com/samuelcolvin/pydantic/blob/5067508eca6222b9315b1e38a30ddc975dee05e7/pydantic/env_settings.py#L26-L31), everything happens inside `__init__`: `_build_environ` is run first, which decodes the banana json into a dict, and then they are stitched together with the `deep_update` function.  

Moving the json decoding into a validator breaks this functionality because it runs after `__init__`. Because the banana mapping is still a string when `__init__` runs, `deep_update` isn't able to merge them and instead replaces the banana dict with the apple dict. 

## Option A: Drop support for merging nested objects

One option would be to drop support for merging nested objects. Complex environment variables seems like an unusual use case to start with, and needing to merge partially specified ones from different sources seems even more unusual. 

I discussed this with @pjbull and this is what we like the most: it would simplify the code and remove the need for a deep update (a shallow update would be enough). 

## Option B: Merging input streams after the universal decoding validator

Currently input streams are merged in `__init__` before running the validators. We could try an approach where the universal validator that does decoding runs on each of the input streams first, and then they get merged after. 

This doesn't really fit into the existing flow very neatly and would involve making some part of the validation flow more complex.

## Option C: Keeping decoding in _build_environ and use a different approach

Such as the approaches that @pjbull brainstormed. 
I faced this issue today, as well. I wanted to parse something like this

```
REDIS_SENTINELS=192.168.0.1 192.168.0.2
```

using such a settings class:

```
class S(BaseSettings):
   sentinels: List[str] = Field(..., env='REDIS_SENTINELS')

    @validator('sentinels', pre=True)
    def validate(cls, val):
        return val.split(' ')
```
Ditto, I'd like to validate a string env var and transform it into a valid submodel. For example, `MY_DB=user:pass@server?max_connections=100` -> a settings model containing a DB submodel with valid DSN and other settings. Currently, it seems I can only pass MY_DB as a stringified JSON object.
Given the complexity of these potential use cases, I'm kind of liking something like proposal (1) in the first comment in this thread. I may have some time on Friday to implement if that approach is interesting
hi all,
I'm also faced with this issue today.

taking into account [this example](https://github.com/tiangolo/full-stack-fastapi-postgresql/blob/0.5.0/%7B%7Bcookiecutter.project_slug%7D%7D/backend/app/app/core/config.py), after almost a day of digging I realized that validator is not fired in case the class attribute is of List-type. If attribute type is str, for example, the attribute validator works just fine.

I used this code:

```python
import os
from typing import List
from pydantic import BaseSettings, validator

os.environ['test_var'] = 'test_val'


class S1(BaseSettings):
    test_var: str
    
    @validator('test_var', pre=True)
    def val_func(cls, v):
        print('this validator is called: {}'.format(v))
        return v

class S2(BaseSettings):
    test_var: List[str]

    @validator('test_var', pre=True)
    def val_func(cls, v):
        print('this validator is called: {}'.format(v))
        return [v]

```

and then instantiating `s1 = S1()` prints `this validator is called: test_val` while the code `s2 = S2()` throws errors and prints nothing:

```
>>> s2 = S2()
Traceback (most recent call last):
  File "pydantic/env_settings.py", line 118, in pydantic.env_settings.BaseSettings._build_environ
  File "/home/den/anaconda3/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/den/anaconda3/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/den/anaconda3/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "pydantic/env_settings.py", line 35, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/env_settings.py", line 48, in pydantic.env_settings.BaseSettings._build_values
  File "pydantic/env_settings.py", line 120, in pydantic.env_settings.BaseSettings._build_environ
pydantic.env_settings.SettingsError: error parsing JSON for "test_var"

```
is there any errors in my example code?
@zdens per some of the discussion earlier in the thread (kind of mixed in there with proposed changes), the reason you don't see the validator firing is because the code that is failing is the part that parses the environment variables, and that happens before the validators run. 

Your variable that is the list needs to be valid JSON, like this:

```
OPTIONS='["a","b","c"]'
```

That's why you see a `JSONDecodeError`.
@jayqi ,
thank you for your reply!
now I understood why this thread has been started.
As a workaround (until something solid is implemented) I changed the following

```python3
CORS_ORIGINS: List[AnyHttpUrl] = Field(..., env="CORS_ORIGINS")
```

into this:

```python3
CORS_ORIGINS: Union[str, List[AnyHttpUrl]] = Field(..., env="CORS_ORIGINS")
```

Which, admittedly, is not elegant but enables pydantic to fire up the validator function:

```python3
@validator("CORS_ORIGINS", pre=True)
    def _assemble_cors_origins(cls, cors_origins):
        if isinstance(cors_origins, str):
            return [item.strip() for item in cors_origins.split(",")]
        return cors_origins
```

My `.env` file:

```env
# Use comma (,) to separate URLs
CORS_ORIGINS=http://localhost:3000,http://localhost:8000
```
I've just discovered how annoying this can be myself while working on a github action for pydantic 🙈.

I'd love someone to fix this, otherwise I will soon.
I planned on working on https://github.com/samuelcolvin/pydantic/pull/1848 again soon (re-implement/finish it), but after finishing with https://github.com/samuelcolvin/pydantic/pull/2721
🎉 I think that if #1848 goes in then we can check for those types here:
https://github.com/samuelcolvin/pydantic/blob/42395056e18dfaa3ef299373374ab3b12bb196ac/pydantic/env_settings.py#L170-L174

And then we can keep the JSON fallback behavior as well.
I suggest that this issue should also consider adding the ability to parse environment variables into dictionaries directly:

```python
import os
from typing import Optional, Dict
from pydantic import BaseSettings, Field, validator

os.environ["COLORS"] = "red:#FF0000,blue:#0000FF,green:#00FF00"


class Example(BaseSettings):
    colors: Optional[Dict[str, str]] = Field(None, env="COLORS")

    # Result would be:
    colors = {"red": "#FF0000", "blue": "#0000FF", "green": "#00FF00"}

```


It seems with the addition of `SecretsSettingsSource` the parsing code got a little bit duplicate. 

I think that now it makes more sense to follow Option 1 described by @pjbull, but also to delegate the whole parsing logic to the Field, i.e: `field.parse(value)`, instead of having the following code in all sources
```
if field.is_complex():
          try:
              secret_value = settings.__config__.json_loads(secret_value)
          except ValueError as e:
              raise SettingsError(f'error parsing JSON for "{env_name}"') from e
```

I could try this implementation if desired @samuelcolvin.

Resolving patch:
diff --git a/docs/examples/settings_with_custom_parsing.py b/docs/examples/settings_with_custom_parsing.py
new file mode 100644
--- /dev/null
+++ b/docs/examples/settings_with_custom_parsing.py
@@ -0,0 +1,19 @@
+import os
+from typing import Any, List
+
+from pydantic import BaseSettings
+
+
+class Settings(BaseSettings):
+    numbers: List[int]
+
+    class Config:
+        @classmethod
+        def parse_env_var(cls, field_name: str, raw_val: str) -> Any:
+            if field_name == 'numbers':
+                return [int(x) for x in raw_val.split(',')]
+            return cls.json_loads(raw_val)
+
+
+os.environ['numbers'] = '1,2,3'
+print(Settings().dict())
diff --git a/pydantic/env_settings.py b/pydantic/env_settings.py
--- a/pydantic/env_settings.py
+++ b/pydantic/env_settings.py
@@ -126,6 +126,10 @@ def customise_sources(
         ) -> Tuple[SettingsSourceCallable, ...]:
             return init_settings, env_settings, file_secret_settings
 
+        @classmethod
+        def parse_env_var(cls, field_name: str, raw_val: str) -> Any:
+            return cls.json_loads(raw_val)
+
     # populated by the metaclass using the Config class defined above, annotated here to help IDEs only
     __config__: ClassVar[Type[Config]]
 
@@ -180,7 +184,7 @@ def __call__(self, settings: BaseSettings) -> Dict[str, Any]:  # noqa C901
                 if env_val is not None:
                     break
 
-            is_complex, allow_json_failure = self.field_is_complex(field)
+            is_complex, allow_parse_failure = self.field_is_complex(field)
             if is_complex:
                 if env_val is None:
                     # field is complex but no value found so far, try explode_env_vars
@@ -190,10 +194,10 @@ def __call__(self, settings: BaseSettings) -> Dict[str, Any]:  # noqa C901
                 else:
                     # field is complex and there's a value, decode that as JSON, then add explode_env_vars
                     try:
-                        env_val = settings.__config__.json_loads(env_val)
+                        env_val = settings.__config__.parse_env_var(field.name, env_val)
                     except ValueError as e:
-                        if not allow_json_failure:
-                            raise SettingsError(f'error parsing JSON for "{env_name}"') from e
+                        if not allow_parse_failure:
+                            raise SettingsError(f'error parsing env var "{env_name}"') from e
 
                     if isinstance(env_val, dict):
                         d[field.alias] = deep_update(env_val, self.explode_env_vars(field, env_vars))
@@ -228,13 +232,13 @@ def field_is_complex(self, field: ModelField) -> Tuple[bool, bool]:
         Find out if a field is complex, and if so whether JSON errors should be ignored
         """
         if field.is_complex():
-            allow_json_failure = False
+            allow_parse_failure = False
         elif is_union(get_origin(field.type_)) and field.sub_fields and any(f.is_complex() for f in field.sub_fields):
-            allow_json_failure = True
+            allow_parse_failure = True
         else:
             return False, False
 
-        return True, allow_json_failure
+        return True, allow_parse_failure
 
     def explode_env_vars(self, field: ModelField, env_vars: Mapping[str, Optional[str]]) -> Dict[str, Any]:
         """
@@ -299,9 +303,9 @@ def __call__(self, settings: BaseSettings) -> Dict[str, Any]:
                     secret_value = path.read_text().strip()
                     if field.is_complex():
                         try:
-                            secret_value = settings.__config__.json_loads(secret_value)
+                            secret_value = settings.__config__.parse_env_var(field.name, secret_value)
                         except ValueError as e:
-                            raise SettingsError(f'error parsing JSON for "{env_name}"') from e
+                            raise SettingsError(f'error parsing env var "{env_name}"') from e
 
                     secrets[field.alias] = secret_value
                 else:


Test patch:
diff --git a/tests/test_settings.py b/tests/test_settings.py
--- a/tests/test_settings.py
+++ b/tests/test_settings.py
@@ -3,7 +3,7 @@
 import uuid
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import Any, Dict, List, Optional, Set, Tuple, Union
+from typing import Any, Callable, Dict, List, Optional, Set, Tuple, Union
 
 import pytest
 
@@ -221,7 +221,7 @@ def test_set_dict_model(env):
 
 def test_invalid_json(env):
     env.set('apples', '["russet", "granny smith",]')
-    with pytest.raises(SettingsError, match='error parsing JSON for "apples"'):
+    with pytest.raises(SettingsError, match='error parsing env var "apples"'):
         ComplexSettings()
 
 
@@ -1054,7 +1054,7 @@ class Settings(BaseSettings):
         class Config:
             secrets_dir = tmp_path
 
-    with pytest.raises(SettingsError, match='error parsing JSON for "foo"'):
+    with pytest.raises(SettingsError, match='error parsing env var "foo"'):
         Settings()
 
 
@@ -1215,3 +1215,66 @@ def test_builtins_settings_source_repr():
         == "EnvSettingsSource(env_file='.env', env_file_encoding='utf-8', env_nested_delimiter=None)"
     )
     assert repr(SecretsSettingsSource(secrets_dir='/secrets')) == "SecretsSettingsSource(secrets_dir='/secrets')"
+
+
+def _parse_custom_dict(value: str) -> Callable[[str], Dict[int, str]]:
+    """A custom parsing function passed into env parsing test."""
+    res = {}
+    for part in value.split(','):
+        k, v = part.split('=')
+        res[int(k)] = v
+    return res
+
+
+def test_env_setting_source_custom_env_parse(env):
+    class Settings(BaseSettings):
+        top: Dict[int, str]
+
+        class Config:
+            @classmethod
+            def parse_env_var(cls, field_name: str, raw_val: str):
+                if field_name == 'top':
+                    return _parse_custom_dict(raw_val)
+                return cls.json_loads(raw_val)
+
+    with pytest.raises(ValidationError):
+        Settings()
+    env.set('top', '1=apple,2=banana')
+    s = Settings()
+    assert s.top == {1: 'apple', 2: 'banana'}
+
+
+def test_env_settings_source_custom_env_parse_is_bad(env):
+    class Settings(BaseSettings):
+        top: Dict[int, str]
+
+        class Config:
+            @classmethod
+            def parse_env_var(cls, field_name: str, raw_val: str):
+                if field_name == 'top':
+                    return int(raw_val)
+                return cls.json_loads(raw_val)
+
+    env.set('top', '1=apple,2=banana')
+    with pytest.raises(SettingsError, match='error parsing env var "top"'):
+        Settings()
+
+
+def test_secret_settings_source_custom_env_parse(tmp_path):
+    p = tmp_path / 'top'
+    p.write_text('1=apple,2=banana')
+
+    class Settings(BaseSettings):
+        top: Dict[int, str]
+
+        class Config:
+            secrets_dir = tmp_path
+
+            @classmethod
+            def parse_env_var(cls, field_name: str, raw_val: str):
+                if field_name == 'top':
+                    return _parse_custom_dict(raw_val)
+                return cls.json_loads(raw_val)
+
+    s = Settings()
+    assert s.top == {1: 'apple', 2: 'banana'}