Repository: pydantic/pydantic

Issue:
Cannot load JSON from secret files
### Checks

* [x] I added a descriptive title to this issue
* [x] I have searched (google, github) for similar issues and couldn't find anything
* [x] I have read and followed [the docs](https://pydantic-docs.helpmanual.io/) and still think this is a bug

<!-- Sorry to sound so draconian, but every second saved replying to issues is time spend improving pydantic :-) -->

# Bug

Output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())"`:
```
             pydantic version: 1.8.2
            pydantic compiled: True
                 install path: /home/natasha/.local/share/virtualenvs/draperweb3-7iyfNlF4/lib/python3.8/site-packages/pydantic
               python version: 3.8.6 (default, May 27 2021, 13:28:02)  [GCC 10.2.0]
                     platform: Linux-5.11.0-7614-generic-x86_64-with-glibc2.32
     optional deps. installed: ['dotenv', 'email-validator', 'typing-extensions']

```
<!-- or if you're using pydantic prior to v1.3, manually include: OS, python version and pydantic version -->

<!-- Please read the [docs](https://pydantic-docs.helpmanual.io/) and search through issues to
confirm your bug hasn't already been reported. -->

When loading settings values from a secrets file, values encoded in JSON do not get correctly parsed, leading to a ValidationError. After looking through the code, I discovered that `SecretsSettingsSource` simply reads the secret value as text instead of checking if it is a complex type. I'm not sure if this is intended, or if it's a bug, but either way, I think it would be beneficial if it were possible to load JSON from a secret file.


Example:
```
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
cloud_config
  value is not a valid dict (type=type_error.dict)
```

<!-- Where possible please include a self-contained code snippet describing your bug: -->

Minimal example

```py
from pydantic import BaseSettings, BaseModel


class CloudConfig(BaseModel):
    client_id: str
    client_secret: str
    token_url: str
    auth_url: str
    api_url: str


class Settings(BaseSettings):
    cloud_config: CloudConfig

    class Config:
        case_sensitive = False
        secrets_dir = "/run/secrets"

settings = Settings()
print(settings.cloud_config.client_id)
```

This works if I have an environment variable `CLOUD_CONFIG='{"client_id": "...", ...}'`, but not if I have the same string in a secrets file



Comments:
Hi @nat212 
Your minimal example is not complete. Can you add what is actually run?
(I'm surprised your `CloudConfig` is not a `BaseSettings`)
Sure, I've updated it to include a little more detail. The example in the [Settings management docs](https://pydantic-docs.helpmanual.io/usage/settings/) use a `BaseModel` as the submodel, so that is why it is not a `BaseSettings`. Would it be better to make it a `BaseSettings`?
If you look at the example, you can see that the `BaseModel` doesn't contain any secret and it's explain in the comment that you need to stringify a dict to override it.
If you want to write only your `client_secret` into `/run/secrets/client_secret`, `CloudConfig` needs to be an `BaseSettings` and have the right `Config` declared
Yes, the issue is that when using a JSON string as an environment variable, it sets the `cloud_config` model successfully. So the environment variable should get detected by the `Settings` model, and parsed as a JSON string, and used to set the `cloud_config` model. However, if I have a secret called `cloud_config` with the same JSON string, the secret just gets read as plain text, and the `Settings` model complains that it has not received a dict

Patch:
diff --git a/pydantic/env_settings.py b/pydantic/env_settings.py
--- a/pydantic/env_settings.py
+++ b/pydantic/env_settings.py
@@ -218,7 +218,14 @@ def __call__(self, settings: BaseSettings) -> Dict[str, Any]:
             for env_name in field.field_info.extra['env_names']:
                 path = secrets_path / env_name
                 if path.is_file():
-                    secrets[field.alias] = path.read_text().strip()
+                    secret_value = path.read_text().strip()
+                    if field.is_complex():
+                        try:
+                            secret_value = settings.__config__.json_loads(secret_value)
+                        except ValueError as e:
+                            raise SettingsError(f'error parsing JSON for "{env_name}"') from e
+
+                    secrets[field.alias] = secret_value
                 elif path.exists():
                     warnings.warn(
                         f'attempted to load secret file "{path}" but found a {path_type(path)} instead.',


Test patch:
diff --git a/tests/test_settings.py b/tests/test_settings.py
--- a/tests/test_settings.py
+++ b/tests/test_settings.py
@@ -830,6 +830,33 @@ class Config:
     assert Settings().dict() == {'foo': 'http://www.example.com', 'bar': SecretStr('snap')}
 
 
+def test_secrets_path_json(tmp_path):
+    p = tmp_path / 'foo'
+    p.write_text('{"a": "b"}')
+
+    class Settings(BaseSettings):
+        foo: Dict[str, str]
+
+        class Config:
+            secrets_dir = tmp_path
+
+    assert Settings().dict() == {'foo': {'a': 'b'}}
+
+
+def test_secrets_path_invalid_json(tmp_path):
+    p = tmp_path / 'foo'
+    p.write_text('{"a": "b"')
+
+    class Settings(BaseSettings):
+        foo: Dict[str, str]
+
+        class Config:
+            secrets_dir = tmp_path
+
+    with pytest.raises(SettingsError, match='error parsing JSON for "foo"'):
+        Settings()
+
+
 def test_secrets_missing(tmp_path):
     class Settings(BaseSettings):
         foo: str