Repository: napari/napari (Python)

Issue:
environmental vars are ignored if an experimental setting is set (e.g. async is enabled)
### 🐛 Bug Report

If I have async (or autoswap) set (enabled) in Preferences > Experimental (box is checked, settings.yaml: `async_: true` ) then trying to *disable* via environmental variable doesn't work.
The opposite does work, so if the setting is **unset** in the Settings, then setting it via environmental variable is respected and does work.

Edit: setting the value directly, rather than by using the `env` value does work, so you *can* use: `NAPARI_EXPERIMENTAL_ASYNC_=0 napari`

Edit2: But setting two variables from the same area, by full name also doesn't work, e.g. `NAPARI_EXPERIMENTAL_AUTOSWAP_BUFFERS=0 NAPARI_EXPERIMENTAL_ASYNC_=0 napari` will only set the first.

### 💡 Steps to Reproduce

Launch napari and ensure that Preferences > Experimental > Render Images Asynchronously is checked (so True).
restart napari with: `NAPARI_ASYNC=0 napari` 
Load Balls (3D) sample and play with slider quickly to see that async is still on.


### 💡 Expected Behavior

The state of the environmental variable should be respected and should take priority over what I have in the settings, so I that I can quickly and easily disable something while testing or demoing.

### 🌎 Environment

napari: 0.5.0a2.dev533+gc6abeb5d
Platform: macOS-14.2.1-arm64-arm-64bit
System: MacOS 14.2.1
Python: 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:38:07) [Clang 16.0.6 ]
Qt: 6.6.1
PyQt6: 
NumPy: 1.26.3
SciPy: 1.12.0
Dask: 2024.1.1
VisPy: 0.14.1
magicgui: 0.8.1
superqt: 0.6.1
in-n-out: 0.1.9
app-model: 0.2.4
npe2: 0.7.4

OpenGL:
- GL version: 2.1 Metal - 88
- MAX_TEXTURE_SIZE: 16384

Screens:
- screen 1: resolution 2056x1329, scale 2.0
- screen 2: resolution 2560x1440, scale 1.0

Settings path:
- /Users/sobolp/Library/Application Support/napari/napari-dev_36963bf0846bfe99dcf4f566a2cc780b6a723afa/settings.yaml

### 💡 Additional Context

_No response_


Discussion: I *think* I may have spotted a problem with the logic of `nested_env_settings` that might be responsible for this. Need to do some more debugging--really should learn how to use a debugger properly I think... 👀 
I will make a PR once I think I'm on the right track.
Happy to review when you have a solution. I haven't looked much at this part of the code before, but would be happy to learn.
So I had/have a fix, but I think there is more going on, because it's not just experimental: ` NAPARI_THEME=light napari` also doesn't work, even though I think it should because: 
https://github.com/napari/napari/blob/89f8194d3fa4eef620755804806ac69ef684df63/napari/settings/_appearance.py#L15
This should work here:
https://github.com/napari/napari/blob/89f8194d3fa4eef620755804806ac69ef684df63/napari/settings/_base.py#L266-L283
But even with my fix of the logic that makes NAPARI-ASYNC, etc. work that one doesn't.
I need to investigate more.
Interestingly, on main `RDP_EPSILON=1 napari` does work, even though it *doesn't* declare an `env`.
Likewise, `USE_NPE2_ADAPTOR=1 napari `
So something is up with EventedSettings handling I think 
Ok! I've figured some more of it out. 
So `experimental` and `plugins` use napari EventedSettings 
https://github.com/napari/napari/blob/89f8194d3fa4eef620755804806ac69ef684df63/napari/settings/_base.py#L41
which uses pydantic BaseSettings class, which sets `env_names` which can be used as environmental variables as follows:
1. using `env='name'` or if not that then
2. using the name of the field with a prefix
See:
https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/v1/env_settings.py#L94-L119

Eitherway, `env_names` will be set!
And in `settings._base.py` we use:
https://github.com/napari/napari/blob/89f8194d3fa4eef620755804806ac69ef684df63/napari/settings/_base.py#L305-L307
So for any EventedSettings class, all fields will be accessible via `env_names`.
And because we don't specify a prefix in either of `experimental` or `plugins` and the default is '', raw `RDP_EPSILON=1` works.

Meanwhile, `appearance` uses just EventedModel, so the inner subfields don't have `env_names` so fields can only be set as env vars using `NAPARI_APPEARANCE_<field>`, so in this case the `env='napari_theme'` is ignored.

So there are two solutions to this conundrum:
1. use `subf.field_info.extra.get('env')` instead of `env_names`, which will catch `NAPARI_THEME`, `NAPARI_ASYNC`, but not the other EventedSettings fields, which will still work using full path.
2. switch `appearance` to also use EventedSettings, leave `nested_env_settings` as is (just fix the for loop logic), add prefix as needed

I think 2 is better option, because it's less hacky and more consistent with Pydantic?

Edit: the other option is to remove `env='napari_theme'`, which judging by tests:
https://github.com/napari/napari/blob/89f8194d3fa4eef620755804806ac69ef684df63/napari/settings/_tests/test_settings.py#L203-L207
may have been intended? Seems like making `NAPARI_THEME` work was discussed in the refactor #2932 https://github.com/napari/napari/pull/2932#issuecomment-869933665 but it was merged before a consensus was reached.
Only thing I don't understand is why `NAPARI_ASYNC=1` works after `napari --reset`, because the `nested_env_settings` doesn't set it.

Edit: if you add
```
experimental:
  async_: false
```
to settings.yaml then `NAPARI_ASYNC=1` stops working. 
So it is a priority thing, where env vars can set settings but not over-ride settings that are already set.

Edit2: So when unset, `NAPARI_ASYNC=1` works because `experimental` is a Pydantic BaseSettings class (if it were EventedModel like `appearance` it wouldn't work).



Fix (patch):
diff --git a/napari/settings/_base.py b/napari/settings/_base.py
index c4c4d22a4c5..603745cac23 100644
--- a/napari/settings/_base.py
+++ b/napari/settings/_base.py
@@ -320,31 +320,33 @@ def _inner(settings: BaseSettings) -> dict[str, Any]:
                     # otherwise, look for the standard nested env var
                     else:
                         env_val = env_vars.get(f'{env_name}_{subf.name}')
-                        if env_val is not None:
-                            break
 
-                is_complex, all_json_fail = super_eset.field_is_complex(subf)
-                if env_val is not None and is_complex:
-                    try:
-                        env_val = settings.__config__.json_loads(env_val)
-                    except ValueError as e:
-                        if not all_json_fail:
-                            msg = trans._(
-                                'error parsing JSON for "{env_name}"',
-                                deferred=True,
-                                env_name=env_name,
+                    is_complex, all_json_fail = super_eset.field_is_complex(
+                        subf
+                    )
+                    if env_val is not None and is_complex:
+                        try:
+                            env_val = settings.__config__.json_loads(env_val)
+                        except ValueError as e:
+                            if not all_json_fail:
+                                msg = trans._(
+                                    'error parsing JSON for "{env_name}"',
+                                    deferred=True,
+                                    env_name=env_name,
+                                )
+                                raise SettingsError(msg) from e
+
+                        if isinstance(env_val, dict):
+                            explode = super_eset.explode_env_vars(
+                                field, env_vars
                             )
-                            raise SettingsError(msg) from e
-
-                    if isinstance(env_val, dict):
-                        explode = super_eset.explode_env_vars(field, env_vars)
-                        env_val = deep_update(env_val, explode)
+                            env_val = deep_update(env_val, explode)
 
-                # if we found an env var, store it and return it
-                if env_val is not None:
-                    if field.alias not in d:
-                        d[field.alias] = {}
-                    d[field.alias][subf.name] = env_val
+                    # if we found an env var, store it and return it
+                    if env_val is not None:
+                        if field.alias not in d:
+                            d[field.alias] = {}
+                        d[field.alias][subf.name] = env_val
         return d
 
     return _inner