Repository: pydantic/pydantic

Issue:
Unexpected behaviour if the prefix contains the env_nested_delimiter
### Checks

* [x] I added a descriptive title to this issue
* [x] I have searched (google, github) for similar issues and couldn't find anything
* [x] I have read and followed [the docs](https://pydantic-docs.helpmanual.io/) and still think this is a bug

<!-- Sorry to sound so draconian, but every second saved replying to issues is time spend improving pydantic :-) -->

# Bug

Output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())"`:
```
$ python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.9.0
            pydantic compiled: True
                 install path: /usr/local/lib/python3.8/site-packages/pydantic
               python version: 3.8.12 (default, Oct 12 2021, 03:05:25)  [GCC 10.2.1 20210110]
                     platform: Linux-5.13.0-28-generic-x86_64-with-glibc2.2.5
     optional deps. installed: ['dotenv', 'typing-extensions']
```
<!-- or if you're using pydantic prior to v1.3, manually include: OS, python version and pydantic version -->

<!-- Please read the [docs](https://pydantic-docs.helpmanual.io/) and search through issues to
confirm your bug hasn't already been reported. -->

<!-- Where possible please include a self-contained code snippet describing your bug: -->

```py
import os
import pydantic
import pytest


@pytest.fixture
def env():
    original_env = os.environ.copy()
    yield os.environ
    os.environ.clear()
    os.environ.update(original_env)


def test_env_nested_delimiter_in_prefix(env):
    """Test pydantic fails to set value where the prefix has the delimiter.

    Pydantic will break down `settings_sub_value` to
    `{sub: {value: ...}}` and it will pass it to `settings.sub`, so it will fail to
    populate `settings.sub.value` and it will try to set `settings.sub.sub`.
    """

    class SubSettings(pydantic.BaseSettings):
        value: str

    class Settings(pydantic.BaseSettings):
        sub: SubSettings

        class Config:
            env_prefix = "settings_"
            env_nested_delimiter = "_"

    env["settings_sub_value"] = "won't be set"

    expected_error = (
        # settings.sub.value doesn't get set
        r".*sub -> value.*\n?.*value_error\.missing.*\n?"
        # Tries to assign settings.sub.sub
        r".*sub -> sub.*\n?.*value_error\.extra"
    )

    with pytest.raises(pydantic.ValidationError, match=expected_error):
        Settings()


def test_env_nested_delimiter_in_field(env):
    """Test pydantic fails to set value where the field has the delimiter.

    Pydantic will break down `with_delimiter_value` to
    `{delimiter: {value: ...}}` and it will pass it to `settings.with_delimiter`,
    so it will fail to populate `settings.with_delimiter.value` and it will try 
    to set `settings.with_delimiter.delimiter`.
    """

    class SubSettings(pydantic.BaseSettings):
        value: str

    class Settings(pydantic.BaseSettings):
        with_delimiter: SubSettings

        class Config:
            env_nested_delimiter = "_"

    env["with_delimiter_value"] = "won't be set"

    expected_error = (
        # settings.with_delimiter.value doesn't get set
        r".*with_delimiter -> value.*\n?.*value_error\.missing.*\n?"
        # Tries to set settings.with_delimiter.delimiter
        r".*with_delimiter -> delimiter.*\n?.*value_error\.extra"
    )

    with pytest.raises(pydantic.ValidationError, match=expected_error):
        Settings()
```
In my opinion this behaviour is wrong. Pydantic should either support this or raise an error that the class is wrongly configured. But the way it behaves right now is confusing since there's no error of wrong class setup, but the fields don't get set as expected from the environment variables.

Here's a possible solution on adding support for allowing to have the delimiter in both in the prefix and in the field name:

```py
class PrefixFixedEnvSettingsSource(EnvSettingsSource):
    def explode_env_vars(
        self, field: ModelField, env_vars: Mapping[str, Optional[str]]
    ) -> Dict[str, Any]:
        """
        Fix original explode_env_vars which splits the prefix with
        env_nested_delimiter. This implemetation doesn't split the prefix.
        """
        prefixes = [
            f"{env_name}{self.env_nested_delimiter}"
            for env_name in field.field_info.extra["env_names"]
        ]
        result: Dict[str, Any] = {}
        for env_name, env_val in env_vars.items():
            matching_prefixes = [
                prefix for prefix in prefixes if env_name.startswith(prefix)
            ]
            if not matching_prefixes:
                continue
            longest_prefix = max(len(prefix) for prefix in matching_prefixes)
            *keys, last_key = env_name[longest_prefix:].split(self.env_nested_delimiter)
            env_var = result
            for key in keys:
                env_var = env_var.setdefault(key, {})
            env_var[last_key] = env_val

        return result
```
Unexpected behaviour if the prefix contains the env_nested_delimiter
### Checks

* [x] I added a descriptive title to this issue
* [x] I have searched (google, github) for similar issues and couldn't find anything
* [x] I have read and followed [the docs](https://pydantic-docs.helpmanual.io/) and still think this is a bug

<!-- Sorry to sound so draconian, but every second saved replying to issues is time spend improving pydantic :-) -->

# Bug

Output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())"`:
```
$ python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.9.0
            pydantic compiled: True
                 install path: /usr/local/lib/python3.8/site-packages/pydantic
               python version: 3.8.12 (default, Oct 12 2021, 03:05:25)  [GCC 10.2.1 20210110]
                     platform: Linux-5.13.0-28-generic-x86_64-with-glibc2.2.5
     optional deps. installed: ['dotenv', 'typing-extensions']
```
<!-- or if you're using pydantic prior to v1.3, manually include: OS, python version and pydantic version -->

<!-- Please read the [docs](https://pydantic-docs.helpmanual.io/) and search through issues to
confirm your bug hasn't already been reported. -->

<!-- Where possible please include a self-contained code snippet describing your bug: -->

```py
import os
import pydantic
import pytest


@pytest.fixture
def env():
    original_env = os.environ.copy()
    yield os.environ
    os.environ.clear()
    os.environ.update(original_env)


def test_env_nested_delimiter_in_prefix(env):
    """Test pydantic fails to set value where the prefix has the delimiter.

    Pydantic will break down `settings_sub_value` to
    `{sub: {value: ...}}` and it will pass it to `settings.sub`, so it will fail to
    populate `settings.sub.value` and it will try to set `settings.sub.sub`.
    """

    class SubSettings(pydantic.BaseSettings):
        value: str

    class Settings(pydantic.BaseSettings):
        sub: SubSettings

        class Config:
            env_prefix = "settings_"
            env_nested_delimiter = "_"

    env["settings_sub_value"] = "won't be set"

    expected_error = (
        # settings.sub.value doesn't get set
        r".*sub -> value.*\n?.*value_error\.missing.*\n?"
        # Tries to assign settings.sub.sub
        r".*sub -> sub.*\n?.*value_error\.extra"
    )

    with pytest.raises(pydantic.ValidationError, match=expected_error):
        Settings()


def test_env_nested_delimiter_in_field(env):
    """Test pydantic fails to set value where the field has the delimiter.

    Pydantic will break down `with_delimiter_value` to
    `{delimiter: {value: ...}}` and it will pass it to `settings.with_delimiter`,
    so it will fail to populate `settings.with_delimiter.value` and it will try 
    to set `settings.with_delimiter.delimiter`.
    """

    class SubSettings(pydantic.BaseSettings):
        value: str

    class Settings(pydantic.BaseSettings):
        with_delimiter: SubSettings

        class Config:
            env_nested_delimiter = "_"

    env["with_delimiter_value"] = "won't be set"

    expected_error = (
        # settings.with_delimiter.value doesn't get set
        r".*with_delimiter -> value.*\n?.*value_error\.missing.*\n?"
        # Tries to set settings.with_delimiter.delimiter
        r".*with_delimiter -> delimiter.*\n?.*value_error\.extra"
    )

    with pytest.raises(pydantic.ValidationError, match=expected_error):
        Settings()
```
In my opinion this behaviour is wrong. Pydantic should either support this or raise an error that the class is wrongly configured. But the way it behaves right now is confusing since there's no error of wrong class setup, but the fields don't get set as expected from the environment variables.

Here's a possible solution on adding support for allowing to have the delimiter in both in the prefix and in the field name:

```py
class PrefixFixedEnvSettingsSource(EnvSettingsSource):
    def explode_env_vars(
        self, field: ModelField, env_vars: Mapping[str, Optional[str]]
    ) -> Dict[str, Any]:
        """
        Fix original explode_env_vars which splits the prefix with
        env_nested_delimiter. This implemetation doesn't split the prefix.
        """
        prefixes = [
            f"{env_name}{self.env_nested_delimiter}"
            for env_name in field.field_info.extra["env_names"]
        ]
        result: Dict[str, Any] = {}
        for env_name, env_val in env_vars.items():
            matching_prefixes = [
                prefix for prefix in prefixes if env_name.startswith(prefix)
            ]
            if not matching_prefixes:
                continue
            longest_prefix = max(len(prefix) for prefix in matching_prefixes)
            *keys, last_key = env_name[longest_prefix:].split(self.env_nested_delimiter)
            env_var = result
            for key in keys:
                env_var = env_var.setdefault(key, {})
            env_var[last_key] = env_val

        return result
```


Discussion:

Resolving patch:
diff --git a/pydantic/env_settings.py b/pydantic/env_settings.py
--- a/pydantic/env_settings.py
+++ b/pydantic/env_settings.py
@@ -63,6 +63,7 @@ def _build_values(
             env_nested_delimiter=(
                 _env_nested_delimiter if _env_nested_delimiter is not None else self.__config__.env_nested_delimiter
             ),
+            env_prefix_len=len(self.__config__.env_prefix),
         )
         file_secret_settings = SecretsSettingsSource(secrets_dir=_secrets_dir or self.__config__.secrets_dir)
         # Provide a hook to set built-in sources priority and add / remove sources
@@ -142,14 +143,19 @@ def __repr__(self) -> str:
 
 
 class EnvSettingsSource:
-    __slots__ = ('env_file', 'env_file_encoding', 'env_nested_delimiter')
+    __slots__ = ('env_file', 'env_file_encoding', 'env_nested_delimiter', 'env_prefix_len')
 
     def __init__(
-        self, env_file: Optional[StrPath], env_file_encoding: Optional[str], env_nested_delimiter: Optional[str] = None
+        self,
+        env_file: Optional[StrPath],
+        env_file_encoding: Optional[str],
+        env_nested_delimiter: Optional[str] = None,
+        env_prefix_len: int = 0,
     ):
         self.env_file: Optional[StrPath] = env_file
         self.env_file_encoding: Optional[str] = env_file_encoding
         self.env_nested_delimiter: Optional[str] = env_nested_delimiter
+        self.env_prefix_len: int = env_prefix_len
 
     def __call__(self, settings: BaseSettings) -> Dict[str, Any]:  # noqa C901
         """
@@ -228,7 +234,9 @@ def explode_env_vars(self, field: ModelField, env_vars: Mapping[str, Optional[st
         for env_name, env_val in env_vars.items():
             if not any(env_name.startswith(prefix) for prefix in prefixes):
                 continue
-            _, *keys, last_key = env_name.split(self.env_nested_delimiter)
+            # we remove the prefix before splitting in case the prefix has characters in common with the delimiter
+            env_name_without_prefix = env_name[self.env_prefix_len :]
+            _, *keys, last_key = env_name_without_prefix.split(self.env_nested_delimiter)
             env_var = result
             for key in keys:
                 env_var = env_var.setdefault(key, {})


Test patch:
diff --git a/tests/test_settings.py b/tests/test_settings.py
--- a/tests/test_settings.py
+++ b/tests/test_settings.py
@@ -134,6 +134,33 @@ class Config:
     }
 
 
+def test_nested_env_delimiter_with_prefix(env):
+    class Subsettings(BaseSettings):
+        banana: str
+
+    class Settings(BaseSettings):
+        subsettings: Subsettings
+
+        class Config:
+            env_nested_delimiter = '_'
+            env_prefix = 'myprefix_'
+
+    env.set('myprefix_subsettings_banana', 'banana')
+    s = Settings()
+    assert s.subsettings.banana == 'banana'
+
+    class Settings(BaseSettings):
+        subsettings: Subsettings
+
+        class Config:
+            env_nested_delimiter = '_'
+            env_prefix = 'myprefix__'
+
+    env.set('myprefix__subsettings_banana', 'banana')
+    s = Settings()
+    assert s.subsettings.banana == 'banana'
+
+
 def test_nested_env_delimiter_complex_required(env):
     class Cfg(BaseSettings):
         v: str = 'default'