Repository: ESMValGroup/ESMValCore

Issue:
Writing of filled recipe fails with `TypeError: '<' not supported between instances of 'str' and 'tuple'`
**Describe the bug**

The recipe

```yaml
documentation:
  title: Test
  description: Test
  authors:
    - schlund_manuel

preprocessors:
  global_ocean:
    mask_landsea:
      mask_out: land
    area_statistics:
      operator: sum

diagnostics:
  test:
    variables:
      fgco2:
        short_name: fgco2
        preprocessor: global_ocean
        project: CMIP5
        mip: Omon
        exp: esmHistorical
        ensemble: r1i1p1
        additional_datasets:
          - {dataset: CanESM2, start_year: 1960, end_year: 2005}
          - {dataset: IPSL-CM5A-LR, start_year: 2000, end_year: 2000}
    scripts:
      null
```

when ran with this configuration

```yaml
rootpath:
  CMIP5:
    /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ: DKRZ
    /work/bd0854/DATA/ESMValTool2/download: ESGF
```

fails with the following error:

```bash
2024-06-27 11:11:33,746 UTC [2721773] ERROR   Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 533, in run
    fire.Fire(ESMValTool())
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 413, in run
    self._run(recipe, session)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 455, in _run
    process_recipe(recipe_file=recipe, session=session)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 130, in process_recipe
    recipe.run()
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/recipe.py", line 1090, in run
    filled_recipe = self.write_filled_recipe()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/recipe.py", line 1128, in write_filled_recipe
    recipe = datasets_to_recipe(USED_DATASETS, self._raw_recipe)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 341, in datasets_to_recipe
    dataset_recipe = _datasets_to_recipe(datasets)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 83, in _datasets_to_recipe
    recipe = _move_datasets_up(recipe)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 92, in _move_datasets_up
    _move_one_level_up(diagnostic, 'variables', 'additional_datasets')
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 119, in _move_one_level_up
    dataset_mapping[name] = {
                            ^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 120, in <dictcomp>
    _to_frozen(ds): ds
    ^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 105, in _to_frozen
    return tuple(sorted((k, _to_frozen(v)) for k, v in item.items()))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 105, in <genexpr>
    return tuple(sorted((k, _to_frozen(v)) for k, v in item.items()))
                            ^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 103, in _to_frozen
    return tuple(sorted(_to_frozen(elem) for elem in item))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'str' and 'tuple'
2024-06-27 11:11:33,763 UTC [2721773] INFO
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.
```

See also https://github.com/ESMValGroup/ESMValTool/issues/3693#issuecomment-2194410283.

Discussion:
bouweandela: @schlunma I think I may have found a working solution now. Previously I used sorted tuples to make the result hashable and independent of the order in which facets appear. I switched to frozensets now, these are also hashable and independent of the order of the facets and do not need to be sorted.

Patch:
diff --git a/esmvalcore/_recipe/from_datasets.py b/esmvalcore/_recipe/from_datasets.py
index f68bd9e09..76cf368a4 100644
--- a/esmvalcore/_recipe/from_datasets.py
+++ b/esmvalcore/_recipe/from_datasets.py
@@ -5,9 +5,10 @@ from __future__ import annotations
 import itertools
 import logging
 import re
+from collections.abc import Iterable, Mapping, Sequence
 from functools import partial
 from pathlib import Path
-from typing import TYPE_CHECKING, Any, Dict, Iterable, Mapping, Sequence
+from typing import TYPE_CHECKING, Any, Dict
 
 from nested_lookup import nested_delete
 
@@ -102,11 +103,13 @@ def _move_datasets_up(recipe: Recipe) -> Recipe:
 
 
 def _to_frozen(item):
-    """Return a frozen and sorted copy of nested dicts and lists."""
-    if isinstance(item, list):
-        return tuple(sorted(_to_frozen(elem) for elem in item))
-    if isinstance(item, dict):
-        return tuple(sorted((k, _to_frozen(v)) for k, v in item.items()))
+    """Return a frozen copy of nested dicts and lists."""
+    if isinstance(item, str):
+        return item
+    if isinstance(item, Mapping):
+        return frozenset((k, _to_frozen(v)) for k, v in item.items())
+    if isinstance(item, Iterable):
+        return frozenset(_to_frozen(elem) for elem in item)
     return item
 
 


Test patch:
diff --git a/tests/unit/recipe/test_from_datasets.py b/tests/unit/recipe/test_from_datasets.py
index 599119195..bc4c71408 100644
--- a/tests/unit/recipe/test_from_datasets.py
+++ b/tests/unit/recipe/test_from_datasets.py
@@ -23,24 +23,35 @@ def test_to_frozen():
                 "d",
                 "c",
             ],
+            "bb": "dc",
         },
     }
 
     result = _to_frozen(data)
-    expected = (
-        (
-            "a",
+    expected = frozenset(
+        {
             (
-                (
-                    "b",
-                    (
-                        "c",
-                        "d",
-                    ),
+                "a",
+                frozenset(
+                    {
+                        (
+                            "b",
+                            frozenset(
+                                {
+                                    "c",
+                                    "d",
+                                }
+                            ),
+                        ),
+                        (
+                            "bb",
+                            "dc",
+                        ),
+                    }
                 ),
             ),
-        ),
-        ("abc", "x"),
+            ("abc", "x"),
+        }
     )
 
     assert result == expected