Repository: Deltares/hydromt

Issue:
create generic Model class (without abstract methods) in core with child classes for different types of models
The goal of this issue is to 
- be more flexible in our assumptions on the type of model. Now there are some assumptions about that a model has a regular grid
- be able to provide tests and examples directly with the Model class
- support different model types, i.e. GridModel (xarray.raster), LumpedModel (xarray), MeshModel (Ugrid convention - xugrid), NetworkModel

TODO:
- [x] identify regular grid assumptions and relax those. make sure model properties like `crs`,  `res` don't throw an error if there are no staticmaps 
- [x] remove abstract methods in Model class and make simple implementation for read_ and write_ methods using common data formats (e.g. NetCDF and GeoJSON)
- [x] Develop GridModel and LumpedModel child classes. 
- [ ] Implement several general setup methods for GridModel: reclass method based on WflowModel.setup_landuse and DelwaqModel.setup_emmissionmapping, resample method based on DelwaqModel.setup_emmissionraster, and rasterizing method based on DelwaqModel.setup_emmissionvector
- [ ] Implement a general setup zonal statistics method for LumpedModel 


Discussion:

Patch:
diff --git a/docs/changelog.rst b/docs/changelog.rst
index 1a7ccc48..34f64c28 100644
--- a/docs/changelog.rst
+++ b/docs/changelog.rst
@@ -13,6 +13,7 @@ Added
 - New raster method for adding gdal_compliant() attributes to xarray object.
 - Function ``to_datetimeindex`` in available preprocess functions for xr.open_dataset in the data adapter.
 - Function ``remove_duplicates`` in available preprocess functions for xr.open_dataset in the data adapter.
+- Optional formatting for year and month variables in path of data sources.
 
 Changed
 ^^^^^^^
diff --git a/docs/user_guide/data_prepare_cat.rst b/docs/user_guide/data_prepare_cat.rst
index d0c42bd6..f9e98dc1 100644
--- a/docs/user_guide/data_prepare_cat.rst
+++ b/docs/user_guide/data_prepare_cat.rst
@@ -60,7 +60,11 @@ A full list of **data source options** is given below
   Relative paths are combined with the global ``root`` option of the yaml file (if available) or the directory of the yaml file itself. 
   To read multiple files in a single dataset (if supported by the driver) a string glob in the form of ``"path/to/my/files/*.nc"`` can be used.
   The filenames can be further specified with ``{variable}``, ``{year}`` and ``{month}`` keys to limit which files are being read 
-  based on the get_data request in the form of ``"path/to/my/files/{variable}_{year}_{month:02d}.nc"``
+  based on the get_data request in the form of ``"path/to/my/files/{variable}_{year}_{month}.nc"``. 
+  Note that ``month`` is by default *not* zero-padded (e.g. January 2012 is stored as ``"path/to/my/files/{variable}_2012_1.nc"``). 
+  Users can optionally add a formatting string to define how the key should be read. 
+  For example, in a path written as ``"path/to/my/files/{variable}_{year}_{month:02d}.nc"``, 
+  the month always has two digits and is zero-padded for Jan-Sep (e.g. January 2012 is stored as ``"path/to/my/files/{variable}_2012_01.nc"``).
 - **data_type** (required): type of input data. Either *RasterDataset*, *GeoDataset* or *GeoDataFrame*.
 - **crs** (required if missing in the data): EPSG code or WKT string of the reference coordinate system of the data. 
 - **driver** (required): data_type specific driver to read a dataset, see overview below.
diff --git a/hydromt/data_adapter/data_adapter.py b/hydromt/data_adapter/data_adapter.py
index d623e2f9..35c5e352 100644
--- a/hydromt/data_adapter/data_adapter.py
+++ b/hydromt/data_adapter/data_adapter.py
@@ -121,34 +121,51 @@ class DataAdapter(object, metaclass=ABCMeta):
         List:
             list of filenames matching the path pattern given date range and variables
         """
-        yr, mth = "*", "*"
-        vrs = ["*"]
-        dates = [""]
-        fns = []
-        path = str(self.path)
         known_keys = ["year", "month", "variable"]
-        keys = [i[1] for i in Formatter().parse(path) if i[1] is not None]
-        # double unknown keys to escape these when formatting
-        for key in [key for key in keys if key not in known_keys]:
-            path = path.replace("{" + key + "}", "{{" + key + "}}")
+        fns = []
+        keys = []
+        # rebuild path based on arguments and escape unknown keys
+        path = ""
+        for literal_text, key, fmt, _ in Formatter().parse(self.path):
+            path += literal_text
+            if key is None:
+                continue
+            key_str = "{" + f"{key}:{fmt}" + "}" if fmt else "{" + key + "}"
+            # remove unused fields
+            if key in ["year", "month"] and time_tuple is None:
+                path += "*"
+            elif key == "variable" and variables is None:
+                path += "*"
+            # escape unknown fields
+            elif key is not None and key not in known_keys:
+                path = path + "{" + key_str + "}"
+            else:
+                path = path + key_str
+                keys.append(key)
         # resolve dates: month & year keys
-        if time_tuple is not None and "year" in keys:
+        dates, vrs, postfix = [None], [None], ""
+        if time_tuple is not None:
             dt = pd.to_timedelta(self.unit_add.get("time", 0), unit="s")
             trange = pd.to_datetime(list(time_tuple)) - dt
-            freq = "m" if "month" in keys else "a"
+            freq, strf = ("m", "%Y-%m") if "month" in keys else ("a", "%Y")
             dates = pd.period_range(*trange, freq=freq)
+            postfix += "; date range: " + " - ".join([t.strftime(strf) for t in trange])
         # resolve variables
-        if variables is not None and "variable" in keys:
+        if variables is not None:
             mv_inv = {v: k for k, v in self.rename.items()}
             vrs = [mv_inv.get(var, var) for var in variables]
+            postfix += f"; variables: {variables}"
+        # get filenames with glob for all date / variable combinations
+        # FIXME: glob won't work with other than local file systems; use fsspec instead
         for date, var in product(dates, vrs):
-            if hasattr(date, "month"):
-                yr, mth = date.year, date.month
-            path1 = path.format(year=yr, month=mth, variable=var)
-            # FIXME: glob won't work with other than local file systems; use fsspec instead
-            fns.extend(glob.glob(path1))
+            fmt = {}
+            if date is not None:
+                fmt.update(year=date.year, month=date.month)
+            if var is not None:
+                fmt.update(variable=var)
+            fns.extend(glob.glob(path.format(**fmt)))
         if len(fns) == 0:
-            raise FileNotFoundError(f"No such file found: {self.path}")
+            raise FileNotFoundError(f"No such file found: {path}{postfix}")
         return list(set(fns))  # return unique paths
 
     @abstractmethod


Test patch:
diff --git a/tests/test_data_adapter.py b/tests/test_data_adapter.py
index bfa16217..791474d7 100644
--- a/tests/test_data_adapter.py
+++ b/tests/test_data_adapter.py
@@ -82,30 +82,34 @@ def test_resolve_path(tmpdir):
     # create dummy files
     for variable in ["precip", "temp"]:
         for year in [2020, 2021]:
-            with open(
-                join(tmpdir, "{unknown_key}_" + f"{variable}_{year}.nc"), "w"
-            ) as f:
-                f.write("")
+            for month in range(1, 13):
+                with open(
+                    join(
+                        tmpdir, "{unknown_key}_" + f"{variable}_{year}_{month:02d}.nc"
+                    ),
+                    "w",
+                ) as f:
+                    f.write("")
     # create data catalog for these files
     dd = {
         "test": {
             "data_type": "RasterDataset",
             "driver": "netcdf",
-            "path": join(tmpdir, "{unknown_key}_{variable}_{year}.nc"),
+            "path": join(tmpdir, "{unknown_key}_{variable}_{year}_{month:02d}.nc"),
         }
     }
     cat = DataCatalog()
     cat.from_dict(dd)
     # test
-    assert len(cat["test"].resolve_paths()) == 4
-    assert len(cat["test"].resolve_paths(variables=["precip"])) == 2
+    assert len(cat["test"].resolve_paths()) == 48
+    assert len(cat["test"].resolve_paths(variables=["precip"])) == 24
     assert (
         len(
             cat["test"].resolve_paths(
                 variables=["precip"], time_tuple=("2021-03-01", "2021-05-01")
             )
         )
-        == 1
+        == 3
     )
     with pytest.raises(FileNotFoundError, match="No such file found:"):
         cat["test"].resolve_paths(variables=["waves"])