Repository: pydata/xarray

Issue:
open_mfdataset with remote files is broken because of #9687
### What happened?

https://github.com/pydata/xarray/pull/9687

This PR broke open_mfdataset with remote files. The ``_normalize_path_list`` doesn't identify them properly and recurses into the remote file

### What did you expect to happen?

This should continue to work, i.e. exit if p is not a list instead of recursing.

### Minimal Complete Verifiable Example

```Python
from distributed import Client

import s3fs
import xarray as xr
s3 = s3fs.S3FileSystem()

file_list = ['s3://nex-gddp-cmip6/NEX-GDDP-CMIP6/ACCESS-CM2/historical/r1i1p1f1/hurs/hurs_day_ACCESS-CM2_historical_r1i1p1f1_gn_1950.nc']
files = [s3.open(f) for f in file_list]


cc @headtr1ck @dcherian 

if __name__ == "__main__":
    client = Client()
    # Load input NetCDF data files
    # TODO: Reduce explicit settings once https://github.com/pydata/xarray/issues/8778 is completed.
    ds = xr.open_mfdataset(
        files,
        engine="h5netcdf",
        combine="nested",
        concat_dim="time",
        data_vars="minimal",
        coords="minimal",
        compat="override",
        parallel=True,
    )
```


### MVCE confirmation

- [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- [x] Complete example — the example is self-contained, including all data and the text of any traceback.
- [x] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result.
- [x] New issue — a search of GitHub Issues suggests this is not a duplicate.
- [x] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

### Relevant log output

```Python
Traceback (most recent call last):
  File "/Users/patrick/Library/Application Support/JetBrains/PyCharm2024.3/scratches/scratch.py", line 19, in <module>
    ds = xr.open_mfdataset(
         ^^^^^^^^^^^^^^^^^^
  File "/Users/patrick/mambaforge/envs/dask-dev/lib/python3.11/site-packages/xarray/backends/api.py", line 1539, in open_mfdataset
    paths = _find_absolute_paths(paths, engine=engine, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/patrick/mambaforge/envs/dask-dev/lib/python3.11/site-packages/xarray/backends/common.py", line 149, in _find_absolute_paths
    return _normalize_path_list(paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/patrick/mambaforge/envs/dask-dev/lib/python3.11/site-packages/xarray/backends/common.py", line 140, in _normalize_path_list
    return [
           ^
  File "/Users/patrick/mambaforge/envs/dask-dev/lib/python3.11/site-packages/xarray/backends/common.py", line 144, in <listcomp>
    else _normalize_path_list(p)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/patrick/mambaforge/envs/dask-dev/lib/python3.11/site-packages/xarray/backends/common.py", line 140, in _normalize_path_list
    return [
           ^
  File "/Users/patrick/mambaforge/envs/dask-dev/lib/python3.11/site-packages/xarray/backends/common.py", line 144, in <listcomp>
    else _normalize_path_list(p)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/patrick/mambaforge/envs/dask-dev/lib/python3.11/site-packages/xarray/backends/common.py", line 140, in _normalize_path_list
    return [
           ^
  File "/Users/patrick/mambaforge/envs/dask-dev/lib/python3.11/site-packages/xarray/backends/common.py", line 144, in <listcomp>
    else _normalize_path_list(p)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/patrick/mambaforge/envs/dask-dev/lib/python3.11/site-packages/xarray/backends/common.py", line 140, in _normalize_path_list
    return [
           ^
TypeError: 'int' object is not iterable
```


### Anything else we need to know?

_No response_

### Environment

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.11.10 | packaged by conda-forge | (main, Oct 16 2024, 01:26:25) [Clang 17.0.6 ]
python-bits: 64
OS: Darwin
OS-release: 23.4.0
machine: arm64
processor: arm
byteorder: little
LC_ALL: None
LANG: None
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.3
libnetcdf: None

xarray: 2024.10.1.dev51+g864b35a1
pandas: 2.2.3
numpy: 2.0.2
scipy: 1.14.1
netCDF4: None
pydap: None
h5netcdf: None
h5py: 3.12.1
zarr: 2.18.3
cftime: None
nc_time_axis: None
iris: None
bottleneck: 1.4.2
dask: 2024.11.2+23.g709bad03e
distributed: 2024.11.2
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2024.10.0
cupy: None
pint: None
sparse: 0.15.4
flox: None
numpy_groupies: None
setuptools: 75.3.0
pip: 24.3.1
conda: None
pytest: 8.3.3
mypy: None
IPython: 8.29.0
sphinx: None
None


</details>


Discussion:
phofl: I added type ignores, mypy should have raised earlier as well since the else clause was incorrect, but type inference was probably a bit smarter here. I really don't know enough about mypy to properly troubleshoot this
phofl: The type hints are generally incorrect in open_mfdataset (I think) since the fsspec version doesn't produce a string or os.PathLike
headtr1ck: > The type hints are generally incorrect in open_mfdataset (I think) since the fsspec version doesn't produce a string or os.PathLike

Do you know what this returns? Is it some buffered reader or similar?
phofl: No, no clue. Nothing is typed over there and there are so many subclasses out there...

The class AbstractFileSystem implements the open but again not really clear

We typed this as

```
ReadBuffer[bytes]
`` in pandas, but this is very generic. The implementation lives here: https://github.com/pandas-dev/pandas/blob/ee3c18f51b393893ed6e31214c7be2f9427ce0c9/pandas/_typing.py#L270
dcherian: OK merging since this fixes a regression. We can followup with typing improvements.

Patch:
diff --git a/xarray/backends/common.py b/xarray/backends/common.py
index c51cf3d1..53456722 100644
--- a/xarray/backends/common.py
+++ b/xarray/backends/common.py
@@ -136,14 +136,15 @@ def _find_absolute_paths(
     def _normalize_path_list(
         lpaths: NestedSequence[str | os.PathLike],
     ) -> NestedSequence[str]:
-        return [
-            (
-                _normalize_path(p)
-                if isinstance(p, str | os.PathLike)
-                else _normalize_path_list(p)
-            )
-            for p in lpaths
-        ]
+        paths = []
+        for p in lpaths:
+            if isinstance(p, str | os.PathLike):
+                paths.append(_normalize_path(p))
+            elif isinstance(p, list):
+                paths.append(_normalize_path_list(p))  # type: ignore[arg-type]
+            else:
+                paths.append(p)  # type: ignore[arg-type]
+        return paths
 
     return _normalize_path_list(paths)
 


Test patch:
diff --git a/xarray/tests/test_backends.py b/xarray/tests/test_backends.py
index 7be6eb5e..c543333c 100644
--- a/xarray/tests/test_backends.py
+++ b/xarray/tests/test_backends.py
@@ -5511,6 +5511,8 @@ def test_source_encoding_always_present_with_fsspec() -> None:
         fs = fsspec.filesystem("file")
         with fs.open(tmp) as f, open_dataset(f) as ds:
             assert ds.encoding["source"] == tmp
+        with fs.open(tmp) as f, open_mfdataset([f]) as ds:
+            assert "foo" in ds
 
 
 def _assert_no_dates_out_of_range_warning(record):