Repository: ecmwf/earthkit-data
Pull request #492 (2024-10-17 20:17:27)

Improve in memory gribfield implementation
This PR:

- reduces the memory usage when a new array fieldlist is created with `FieldList.to_fieldlist()` 
- reduces the memory usage when `to_numpy()`, `to_array()` or `values` called on a fieldlist
- applies the `use-grib-metadata-cache` setting to all types of GRIB fields, even for fields entirely in memory




codecov-commenter: ## [Codecov](https://app.codecov.io/gh/ecmwf/earthkit-data/pull/492?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf) Report
Attention: Patch coverage is `99.18699%` with `1 line` in your changes missing coverage. Please review.
> Project coverage is 90.66%. Comparing base [(`2656f54`)](https://app.codecov.io/gh/ecmwf/earthkit-data/commit/2656f547d9d7adb151bc5f1fa41dc7c2b03c6133?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf) to head [(`38cd7c5`)](https://app.codecov.io/gh/ecmwf/earthkit-data/commit/38cd7c58c1e183b1edc3192af6cab142629dad50?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf).
> Report is 1 commits behind head on develop.

| [Files with missing lines](https://app.codecov.io/gh/ecmwf/earthkit-data/pull/492?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf) | Patch % | Lines |
|---|---|---|
| [tests/grib/grib\_fixtures.py](https://app.codecov.io/gh/ecmwf/earthkit-data/pull/492?src=pr&el=tree&filepath=tests%2Fgrib%2Fgrib_fixtures.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf#diff-dGVzdHMvZ3JpYi9ncmliX2ZpeHR1cmVzLnB5) | 92.30% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/ecmwf/earthkit-data/pull/492?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf) |

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           develop     #492      +/-   ##
===========================================
+ Coverage    90.59%   90.66%   +0.06%     
===========================================
  Files          143      143              
  Lines         9755     9821      +66     
  Branches       468      472       +4     
===========================================
+ Hits          8838     8904      +66     
  Misses         761      761              
  Partials       156      156              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/ecmwf/earthkit-data/pull/492?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf).


Patch:
diff --git a/docs/examples/numpy_fieldlist.ipynb b/docs/examples/numpy_fieldlist.ipynb
index 92beb62..f80b849 100644
--- a/docs/examples/numpy_fieldlist.ipynb
+++ b/docs/examples/numpy_fieldlist.ipynb
@@ -57,7 +57,7 @@
    "id": "9e6a80f5-77fc-40f8-a0be-86b80275c390",
    "metadata": {},
    "source": [
-    "In the examples below we will compute the potential temperature using the following method:"
+    "We will use the following method to compute the potential temperature:"
    ]
   },
   {
@@ -98,7 +98,7 @@
     {
      "data": {
       "text/plain": [
-       "GribField(t,850,20180801,1200,0,0)"
+       "(GribField(t,850,20180801,1200,0,0), 'K')"
       ]
      },
      "execution_count": 3,
@@ -108,51 +108,7 @@
    ],
    "source": [
     "f = ds[3]\n",
-    "f"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "0025e432-afa1-40ce-8968-7543364ef5f8",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'K'"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "f.metadata(\"units\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "a01201de-4aca-4ada-ab29-474d45d53e90",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "array([272.53916931, 272.53916931, 272.53916931, 272.53916931,\n",
-       "       272.53916931, 272.53916931, 272.53916931, 272.53916931,\n",
-       "       272.53916931, 272.53916931])"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "f.values[:10]"
+    "f, f.metadata(\"units\")"
    ]
   },
   {
@@ -165,7 +121,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 4,
    "id": "60db0219-044d-4977-abf5-e30065400a8f",
    "metadata": {},
    "outputs": [
@@ -184,7 +140,7 @@
        "       285.48786915, 285.48786915])"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -209,12 +165,12 @@
     "tags": []
    },
    "source": [
-    "We create a new :py:class:`~data.readers.grib.metadata.GribMetadata` object from the source field's :py:meth:`~data.readers.grib.codes.GribField.metadata` using :py:meth:`~data.readers.grib.metadata.GribMetadata.override`."
+    "We create a new :py:class:`~data.readers.grib.metadata.GribMetadata` object from the source field's :py:meth:`~data.readers.grib.codes.GribField.metadata` using :py:meth:`~data.readers.grib.metadata.GribMetadata.override`. The original metadata remains the same."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 5,
    "id": "f8dcc33c-6818-415d-8088-6a01666ca87f",
    "metadata": {
     "editable": true,
@@ -223,36 +179,6 @@
     },
     "tags": []
    },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "<earthkit.data.readers.grib.metadata.StandAloneGribMetadata at 0x1694997b0>"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "md_new = f.metadata().override(shortName=\"pt\")\n",
-    "md_new"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "34aec901-4f60-4154-99ac-890172b96efa",
-   "metadata": {},
-   "source": [
-    "The original metadata remains the same."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "67a58bb4-e435-4d01-9e97-5965e0ac49f9",
-   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -264,8 +190,9 @@
     }
    ],
    "source": [
+    "md_new = f.metadata().override(shortName=\"pt\")\n",
     "print(md_new[\"shortName\"])\n",
-    "print(f.metadata()[\"shortName\"])"
+    "print(f.metadata(\"shortName\"))"
    ]
   },
   {
@@ -285,7 +212,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 6,
    "id": "5a2557a6-091c-4345-a792-d3bfa7876c77",
    "metadata": {
     "editable": true,
@@ -354,7 +281,7 @@
        "0       an       0  regular_ll  "
       ]
      },
-     "execution_count": 9,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -366,7 +293,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 7,
    "id": "2ccde1fc-f8c4-4d4d-abda-3ca3feebd02e",
    "metadata": {},
    "outputs": [
@@ -376,7 +303,7 @@
        "ArrayField(pt,850,20180801,1200,0,0)"
       ]
      },
-     "execution_count": 10,
+     "execution_count": 7,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -385,56 +312,6 @@
     "ds_new[0]"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "4113f204-dd96-422d-acb5-e2e0901cfb7a",
-   "metadata": {
-    "editable": true,
-    "slideshow": {
-     "slide_type": ""
-    },
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "array([285.48786915, 285.48786915, 285.48786915, 285.48786915,\n",
-       "       285.48786915, 285.48786915, 285.48786915, 285.48786915,\n",
-       "       285.48786915, 285.48786915])"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "ds_new[0].values[:10]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "cee91d37-2444-477d-b43f-fbd2b58c4519",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Potential temperature'"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "ds_new[0].metadata(\"name\")"
-   ]
-  },
   {
    "cell_type": "raw",
    "id": "c7733bdd-1a47-4dba-aac4-b6ed43e15337",
@@ -447,12 +324,12 @@
     "tags": []
    },
    "source": [
-    "We can save our new fieldlist into a GRIB file:"
+    "The new fieldlist can be saved into a GRIB file:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 8,
    "id": "c22a23fc-fccd-45ef-887e-95ab61b0dee1",
    "metadata": {
     "editable": true,
@@ -521,7 +398,7 @@
        "0       an       0  regular_ll  "
       ]
      },
-     "execution_count": 13,
+     "execution_count": 8,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -529,6 +406,8 @@
    "source": [
     "path = \"_pt_single.grib\"\n",
     "ds_new.save(path)\n",
+    "\n",
+    "# read file back and check content\n",
     "ds1 = earthkit.data.from_source(\"file\",path)\n",
     "ds1.ls()"
    ]
@@ -551,7 +430,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 9,
    "id": "bedb28e0-dc98-4145-8559-da96cadca828",
    "metadata": {},
    "outputs": [
@@ -648,7 +527,7 @@
        "2       an       0  regular_ll     K  "
       ]
      },
-     "execution_count": 14,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -660,7 +539,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 10,
    "id": "74881517-ce58-4692-b1f1-cf281c5c1df2",
    "metadata": {},
    "outputs": [
@@ -670,7 +549,7 @@
        "(3, 84)"
       ]
      },
-     "execution_count": 15,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -681,29 +560,6 @@
     "t_new.shape"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "bedf4a1f-1696-4da9-ae6b-575cd42d692c",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "array([285.48786915, 285.48786915, 285.48786915, 285.48786915,\n",
-       "       285.48786915, 285.48786915, 285.48786915, 285.48786915,\n",
-       "       285.48786915, 285.48786915])"
-      ]
-     },
-     "execution_count": 16,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "t_new[0,:10]"
-   ]
-  },
   {
    "cell_type": "raw",
    "id": "8f2f7612-3ff9-4982-9cb0-a91a426f1b0f",
@@ -721,7 +577,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 11,
    "id": "ab0c805e-9e33-40f1-a816-501c2cff64a1",
    "metadata": {
     "editable": true,
@@ -820,7 +676,7 @@
        "2       an       0  regular_ll  "
       ]
      },
-     "execution_count": 17,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -831,50 +687,6 @@
     "ds_new.ls()"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "b0298472-345e-41f4-836e-a05b45509b2b",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "array([285.48786915, 285.48786915, 285.48786915, 285.48786915,\n",
-       "       285.48786915, 285.48786915, 285.48786915, 285.48786915,\n",
-       "       285.48786915, 285.48786915])"
-      ]
-     },
-     "execution_count": 18,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "ds_new[0].values[:10]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "835412a7-85a2-4b07-ae40-e7e816e1a0bf",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "['Potential temperature', 'Potential temperature', 'Potential temperature']"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "ds_new.metadata(\"name\")"
-   ]
-  },
   {
    "cell_type": "raw",
    "id": "ca785b7b-d7a8-424b-99c5-19056a0b23b0",
@@ -887,12 +699,12 @@
     "tags": []
    },
    "source": [
-    "We can save the fieldlist into a GRIB file:"
+    "The new fieldlist can be saved into a GRIB file:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 12,
    "id": "e8e1c8a2-87aa-4ec2-895e-8ea7bd83e101",
    "metadata": {
     "editable": true,
@@ -991,7 +803,7 @@
        "2       an       0  regular_ll  "
       ]
      },
-     "execution_count": 20,
+     "execution_count": 12,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -999,6 +811,8 @@
    "source": [
     "path = \"_pt_multi.grib\"\n",
     "ds_new.save(path)\n",
+    "\n",
+    "# read file back and check content\n",
     "ds1 = earthkit.data.from_source(\"file\", path)\n",
     "ds1.ls()"
    ]
@@ -1028,7 +842,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 13,
    "id": "6c6e8651-f5a3-4534-9227-812e8e69407d",
    "metadata": {
     "editable": true,
@@ -1058,7 +872,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 14,
    "id": "32c53494-e8f4-4a22-aa93-e16b5188483a",
    "metadata": {},
    "outputs": [
@@ -1151,7 +965,7 @@
        "2       an       0  regular_ll  "
       ]
      },
-     "execution_count": 22,
+     "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1175,12 +989,12 @@
     }
    },
    "source": [
-    "We can save the results into a GRIB file:"
+    "The new fieldlist can be saved into a GRIB file:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 15,
    "id": "a149caa0-b1ba-4b8e-b2d0-6aad7ee224a6",
    "metadata": {},
    "outputs": [
@@ -1273,7 +1087,7 @@
        "2       an       0  regular_ll  "
       ]
      },
-     "execution_count": 23,
+     "execution_count": 15,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1281,6 +1095,8 @@
    "source": [
     "path = \"_pt_from_loop.grib\"\n",
     "ds_r.save(path)\n",
+    "\n",
+    "# read file back and check content\n",
     "ds1 = earthkit.data.from_source(\"file\", path)\n",
     "ds1.ls()"
    ]
diff --git a/docs/guide/misc/grib_memory.rst b/docs/guide/misc/grib_memory.rst
index 9c25847..417da01 100644
--- a/docs/guide/misc/grib_memory.rst
+++ b/docs/guide/misc/grib_memory.rst
@@ -45,7 +45,6 @@ When reading :ref:`grib` data from disk as a :ref:`file source <data-sources-fil
 - :ref:`grib-field-policy <grib-field-policy>`
 - :ref:`grib-handle-policy <grib-handle-policy>`
 - :ref:`grib-handle-cache-size <grib-handle-cache-size>`
-- :ref:`use-grib-metadata-cache <use-grib-metadata-cache>`
 
 .. _grib-field-policy:
 
@@ -79,18 +78,10 @@ grib-handle-cache-size
 
 When :ref:`grib-handle-policy <grib-handle-policy>` is ``"cache"``, the setting ``grib-handle-cache-size`` (default is ``1``) specifies the maximum number of GRIB handles kept in an in-memory cache per fieldlist. This is an LRU cache, so when it is full, the least recently used GRIB handle is removed and a new GRIB message is loaded from disk and added to the cache.
 
-.. _use-grib-metadata-cache:
-
-use-grib-metadata-cache
-+++++++++++++++++++++++++++++++++++
-
-When ``use-grib-metadata-cache`` is ``True`` (this is the default) all the fields will cache their metadata access. This is an in-memory cache attached to the field and implemented for the low-level metadata accessor for individual keys. This cache can be useful when the same metadata keys are accessed multiple times for the same field.
-
-
 Overriding the settings
 ++++++++++++++++++++++++++++
 
-In addition to changing the :ref:`settings` themselves, it is possible to override the 4 parameters above when loading a given fieldlist by passing them as keyword arguments to :func:`from_source`. The parameter names are the same but the dashes are replaced by underscores. When a parameter is not specified in :func:`from_source` or is set to None, its value is taken from the actual :ref:`settings`. E.g.:
+In addition to changing the :ref:`settings` themselves, it is possible to override the parameters above when loading a given fieldlist by passing them as keyword arguments to :func:`from_source`. The parameter names are the same but the dashes are replaced by underscores. When a parameter is not specified in :func:`from_source` or is set to None, its value is taken from the actual :ref:`settings`. E.g.:
 
 .. code-block:: python
 
@@ -102,7 +93,6 @@ In addition to changing the :ref:`settings` themselves, it is possible to overri
         grib_field_policy="persistent",
         grib_handle_policy="temporary",
         grib_handle_cache_size=0,
-        use_grib_metadata_cache=True,
     )
 
 
diff --git a/docs/guide/misc/grib_metadata.rst b/docs/guide/misc/grib_metadata.rst
new file mode 100644
index 0000000..fd86ae1
--- /dev/null
+++ b/docs/guide/misc/grib_metadata.rst
@@ -0,0 +1,27 @@
+.. _grib-metadata-cache
+
+GRIB field metadata caching
+//////////////////////////////
+
+The ``use-grib-metadata-cache`` :ref:`setting <settings>` controls whether :ref:`grib` fields will cache their metadata access. The default value is ``True``.
+
+This is an in-memory cache attached to the field and implemented for the low-level metadata accessor for individual keys. Getting the values from the cache can be significantly faster than reading them from the GRIB handle, even when the handle is kept in memory.
+
+This setting is applied to all the different GRIB field types, even for fields stored entirely in memory (see :ref:`grib-memory`).
+
+
+Overriding the settings
+++++++++++++++++++++++++++++
+
+In addition to changing the :ref:`settings`, it is possible to override ``use-grib-metadata-cache`` when loading a given fieldlist by passing the ``use_grib_metadata_cache`` keyword argument (note the underscores) to :func:`from_source`. When this kwarg is not specified in :func:`from_source` or is set to None, its value is taken from the actual :ref:`settings`. E.g.:
+
+.. code-block:: python
+
+    import earthkit.data
+
+    # will override the settings
+    ds = earthkit.data.from_source(
+        "file",
+        "test6.grib",
+        use_grib_metadata_cache=False,
+    )
diff --git a/src/earthkit/data/core/fieldlist.py b/src/earthkit/data/core/fieldlist.py
index ad268fb..3adf33c 100644
--- a/src/earthkit/data/core/fieldlist.py
+++ b/src/earthkit/data/core/fieldlist.py
@@ -705,6 +705,27 @@ class Field(Base):
 
         # return {name: metadata(name) for name in names}
 
+    def to_field(self, array_backend=None, **kwargs):
+        r"""Convert to a new :class:`Field`.
+
+        Parameters
+        ----------
+        array_backend: str, module, :obj:`ArrayBackend`
+            Specifies the array backend for the generated :class:`Field`. The array
+            type must be supported by :class:`ArrayBackend`.
+
+        **kwargs: dict, optional
+            ``kwargs`` are passed to :obj:`to_array` to
+            extract the field values the resulting object will store.
+
+        Returns
+        -------
+        :class:`ArrayField`
+        """
+        from earthkit.data.sources.array_list import ArrayField
+
+        return ArrayField(self.to_array(array_backend=array_backend, **kwargs), self._metadata.override())
+
     @staticmethod
     def _flatten(v):
         """Flatten the array without copying the data."
@@ -893,6 +914,27 @@ class FieldList(Index):
         """
         return self._md_indices.index(key)
 
+    def _as_array(self, accessor, **kwargs):
+        """Use pre-allocated target array to store the field values."""
+
+        def _vals(f):
+            return getattr(f, accessor)(**kwargs) if not is_property else getattr(f, accessor)
+
+        n = len(self)
+        if n > 0:
+            it = iter(self)
+            first = next(it)
+            is_property = isinstance(getattr(first.__class__, accessor), property)
+            vals = _vals(first)
+            ns = array_namespace(vals)
+            shape = (n, *vals.shape)
+            r = ns.empty(shape, dtype=vals.dtype)
+            r[0] = vals
+            for i, f in enumerate(it, start=1):
+                r[i] = _vals(f)
+
+            return r
+
     def to_numpy(self, **kwargs):
         r"""Return all the fields' values as an ndarray. It is formed as the array of the
         :obj:`data.core.fieldlist.Field.to_numpy` values per field.
@@ -912,9 +954,7 @@ class FieldList(Index):
         to_array
         values
         """
-        import numpy as np
-
-        return np.array([f.to_numpy(**kwargs) for f in self])
+        return self._as_array("to_numpy", **kwargs)
 
     def to_array(self, **kwargs):
         r"""Return all the fields' values as an array. It is formed as the array of the
@@ -935,10 +975,7 @@ class FieldList(Index):
         values
         to_numpy
         """
-        x = [f.to_array(**kwargs) for f in self]
-        if len(x) == 0:
-            return None
-        return array_namespace(x[0]).stack(x)
+        return self._as_array("to_array", **kwargs)
 
     @property
     def values(self):
@@ -963,10 +1000,7 @@ class FieldList(Index):
         array([262.78027344, 267.44726562, 268.61230469])
 
         """
-        x = [f.values for f in self]
-        if len(x) == 0:
-            return None
-        return array_namespace(x[0]).stack(x)
+        return self._as_array("values")
 
     def data(
         self,
@@ -1471,9 +1505,6 @@ class FieldList(Index):
     def to_fieldlist(self, array_backend=None, **kwargs):
         r"""Convert to a new :class:`FieldList`.
 
-        When the :class:`FieldList` is already in the required format no new
-        :class:`FieldList` is created but the current one is returned.
-
         Parameters
         ----------
         array_backend: str, module, :obj:`ArrayBackend`
@@ -1486,9 +1517,8 @@ class FieldList(Index):
 
         Returns
         -------
-        :class:`FieldList`
-            - the current :class:`FieldList` if it is already in the required format
-            - a new :class:`SimpleFieldList` with :class`ArrayField` fields otherwise
+        :class:`SimpleFieldList`
+            - a new fieldlist containing :class`ArrayField` fields
 
         Examples
         --------
@@ -1509,12 +1539,7 @@ class FieldList(Index):
         dtype('float32')
 
         """
-        array = []
-        md = []
-        for f in self:
-            array.append(f.to_array(array_backend=array_backend, **kwargs))
-            md.append(f._metadata)
-        return self.from_array(array, md)
+        return self.from_fields([f.to_field(array_backend=array_backend, **kwargs) for f in self])
 
     def cube(self, *args, **kwargs):
         from earthkit.data.indexing.cube import FieldCube
@@ -1530,6 +1555,12 @@ class FieldList(Index):
         assert all(isinstance(_, FieldList) for _ in sources)
         return MultiFieldList(sources)
 
+    def _cache_diag(self):
+        """For testing only"""
+        from earthkit.data.utils.diag import metadata_cache_diag
+
+        return metadata_cache_diag(self)
+
 
 class MaskFieldList(FieldList, MaskIndex):
     def __init__(self, *args, **kwargs):
diff --git a/src/earthkit/data/indexing/fieldlist.py b/src/earthkit/data/indexing/fieldlist.py
index f5acb4a..c3c5170 100644
--- a/src/earthkit/data/indexing/fieldlist.py
+++ b/src/earthkit/data/indexing/fieldlist.py
@@ -7,6 +7,7 @@
 # nor does it submit to any jurisdiction.
 #
 
+
 from earthkit.data.core.fieldlist import FieldList
 
 
diff --git a/src/earthkit/data/readers/grib/codes.py b/src/earthkit/data/readers/grib/codes.py
index 55efdc2..7cbf014 100644
--- a/src/earthkit/data/readers/grib/codes.py
+++ b/src/earthkit/data/readers/grib/codes.py
@@ -9,7 +9,6 @@
 
 import logging
 import os
-from collections import defaultdict
 from functools import cached_property
 
 import eccodes
@@ -329,14 +328,3 @@ class GribField(Field):
         bytes
         """
         return self.handle.get_buffer()
-
-    def _diag(self):
-        r = r = defaultdict(int)
-        try:
-            md_cache = self._metadata._cache
-            r["metadata_cache_size"] += len(md_cache)
-            r["metadata_cache_hits"] += md_cache.hits
-            r["metadata_cache_misses"] += md_cache.misses
-        except Exception:
-            pass
-        return r
diff --git a/src/earthkit/data/readers/grib/index/__init__.py b/src/earthkit/data/readers/grib/index/__init__.py
index 1e381cf..11a881c 100644
--- a/src/earthkit/data/readers/grib/index/__init__.py
+++ b/src/earthkit/data/readers/grib/index/__init__.py
@@ -394,23 +394,21 @@ class GribFieldListInFiles(GribFieldList):
     def __len__(self):
         return self.number_of_parts()
 
-    def _diag(self):
+    def _cache_diag(self):
+        """For testing only"""
         r = defaultdict(int)
         r.update(self._field_manager.diag())
         r.update(self._handle_manager.diag())
 
         if self._field_manager.cache is not None:
+            from earthkit.data.utils.diag import collect_field_metadata_cache_diag
+
             for f in self._field_manager.cache.values():
                 if f._handle is not None:
                     r["current_handle_count"] += 1
 
                 if self._use_metadata_cache:
-                    try:
-                        md_cache = f._diag()
-                        for k in ["metadata_cache_hits", "metadata_cache_misses", "metadata_cache_size"]:
-                            r[k] += md_cache[k]
-                    except Exception:
-                        pass
+                    collect_field_metadata_cache_diag(f, r)
         return r
 
     @abstractmethod
diff --git a/src/earthkit/data/readers/grib/memory.py b/src/earthkit/data/readers/grib/memory.py
index df13eaf..8f43d0b 100644
--- a/src/earthkit/data/readers/grib/memory.py
+++ b/src/earthkit/data/readers/grib/memory.py
@@ -8,7 +8,6 @@
 #
 
 import logging
-from functools import cached_property
 
 import eccodes
 
@@ -16,14 +15,23 @@ from earthkit.data.indexing.fieldlist import SimpleFieldList
 from earthkit.data.readers import Reader
 from earthkit.data.readers.grib.codes import GribCodesHandle
 from earthkit.data.readers.grib.codes import GribField
-from earthkit.data.readers.grib.metadata import GribFieldMetadata
 
 LOG = logging.getLogger(__name__)
 
 
+def get_use_grib_metadata_cache():
+    from earthkit.data.core.settings import SETTINGS
+
+    return SETTINGS.get("use-grib-metadata-cache")
+
+
 class GribMemoryReader(Reader):
-    def __init__(self, **kwargs):
+    def __init__(self, use_grib_metadata_cache=None, **kwargs):
         self._peeked = None
+        self.consumed_ = False
+        self._use_metadata_cache = (
+            use_grib_metadata_cache if use_grib_metadata_cache is not None else get_use_grib_metadata_cache()
+        )
 
     def __iter__(self):
         return self
@@ -45,7 +53,9 @@ class GribMemoryReader(Reader):
 
     def _message_from_handle(self, handle):
         if handle is not None:
-            return GribFieldInMemory(GribCodesHandle(handle, None, None))
+            return GribFieldInMemory(
+                GribCodesHandle(handle, None, None), use_metadata_cache=self._use_metadata_cache
+            )
 
     def batched(self, n):
         from earthkit.data.utils.batch import batched
@@ -117,8 +127,8 @@ class GribStreamReader(GribMemoryReader):
 class GribFieldInMemory(GribField):
     """Represents a GRIB message in memory"""
 
-    def __init__(self, handle):
-        super().__init__(None, None, None)
+    def __init__(self, handle, use_metadata_cache=False):
+        super().__init__(None, None, None, use_metadata_cache=use_metadata_cache)
         self._handle = handle
 
     @GribField.handle.getter
@@ -129,9 +139,9 @@ class GribFieldInMemory(GribField):
     def offset(self):
         return None
 
-    @cached_property
-    def _metadata(self):
-        return GribFieldMetadata(self)
+    # @cached_property
+    # def _metadata(self):
+    #     return GribFieldMetadata(self)
 
     @staticmethod
     def to_fieldlist(fields):
@@ -140,21 +150,14 @@ class GribFieldInMemory(GribField):
     @staticmethod
     def from_buffer(buf):
         handle = eccodes.codes_new_from_message(buf)
-        return GribFieldInMemory(GribCodesHandle(handle, None, None))
+        return GribFieldInMemory(
+            GribCodesHandle(handle, None, None), use_metadata_cache=get_use_grib_metadata_cache()
+        )
 
 
 class GribFieldListInMemory(SimpleFieldList):
     """Represent a GRIB field list in memory loaded lazily"""
 
-    # @staticmethod
-    # def from_fields(fields):
-    #     if array_backend is None and len(fields) > 0:
-    #         array_backend = fields[0].array_backend
-    #     fs = GribFieldListInMemory(None, None, array_backend=array_backend)
-    #     fs.fields = fields
-    #     fs._loaded = True
-    #     return fs
-
     def __init__(self, source, reader, *args, **kwargs):
         """The reader must support __next__."""
         if source is not None:
diff --git a/src/earthkit/data/readers/grib/metadata.py b/src/earthkit/data/readers/grib/metadata.py
index 3ab563c..cdeeb24 100644
--- a/src/earthkit/data/readers/grib/metadata.py
+++ b/src/earthkit/data/readers/grib/metadata.py
@@ -408,7 +408,12 @@ class GribMetadata(Metadata):
             vals = np.zeros(new_value_size)
             handle.set_values(vals)
 
-        return StandAloneGribMetadata(handle)
+        # ensure that the cache settings are the same
+        cache = self._cache
+        if cache is not None:
+            cache = cache.__class__()
+
+        return StandAloneGribMetadata(handle, cache=cache)
 
     def as_namespace(self, namespace=None):
         r"""Return all the keys/values from a namespace.
@@ -588,12 +593,19 @@ class StandAloneGribMetadata(GribMetadata):
     def __getstate__(self) -> dict:
         ret = {}
         ret["_handle"] = self._handle.get_buffer()
+        # we do not serialize the cache contents
+        ret["_cache"] = self._cache if self._cache is None else self._cache.__class__
         return ret
 
     def __setstate__(self, state: dict):
         from earthkit.data.readers.grib.memory import GribMessageMemoryReader
 
+        cache = state.pop("_cache")
+        if cache is not None:
+            cache = cache()
+
         self.__handle = next(GribMessageMemoryReader(state.pop("_handle"))).handle
+        self._cache = cache
 
 
 class RestrictedGribMetadata(WrappedMetadata):
diff --git a/src/earthkit/data/sources/array_list.py b/src/earthkit/data/sources/array_list.py
index 6695a18..0d99c06 100644
--- a/src/earthkit/data/sources/array_list.py
+++ b/src/earthkit/data/sources/array_list.py
@@ -38,6 +38,7 @@ class ArrayField(Field):
 
             metadata = UserMetadata(metadata, values=array)
 
+        # TODO: this solution is questionable due to performance reasons
         metadata = metadata._hide_internal_keys()
 
         super().__init__(metadata=metadata)
diff --git a/src/earthkit/data/utils/diag.py b/src/earthkit/data/utils/diag.py
index 936e6c6..ee48238 100644
--- a/src/earthkit/data/utils/diag.py
+++ b/src/earthkit/data/utils/diag.py
@@ -8,6 +8,7 @@
 #
 
 import time
+from collections import defaultdict
 
 
 class TimeDiag:
@@ -19,7 +20,7 @@ class TimeDiag:
     def elapsed(self):
         return time.time() - self.start
 
-    def __call__(self, label=""):
+    def __call__(self, label="", as_str=False):
         curr = time.time()
         delta = curr - self.prev
         self.prev = curr
@@ -31,11 +32,16 @@ class TimeDiag:
             else:
                 label = f"[{self.name}]"
 
-        print(f"{label} elapsed={self.elapsed():.3f} delta={delta:.3f}")
+        s = f"{label}elapsed={self.elapsed():.3f}s delta={delta:.3f}s"
+
+        if as_str:
+            return s
+        else:
+            print(s)
 
 
 class MemoryDiag:
-    def __init__(self, name=""):
+    def __init__(self, name="", peak=False):
         import os
         import platform
 
@@ -45,6 +51,7 @@ class MemoryDiag:
         self.proc = psutil.Process()
         self.prev = 0
         self.scale = 1
+        self.add_peak = peak
 
         try:
             if os.name == "posix" and platform.system() == "Darwin":
@@ -74,7 +81,7 @@ class MemoryDiag:
         rss = getrusage(RUSAGE_SELF).ru_maxrss
         return self.scale_to_mbytes(rss)
 
-    def __call__(self, label="", delta=True):
+    def __call__(self, label="", delta=True, as_str=False):
         m = self.current()
         _delta = m - self.prev
         self.prev = m
@@ -87,17 +94,64 @@ class MemoryDiag:
             else:
                 label = f"[{self.name}]"
 
+        s = ""
         if delta:
-            print(f"{label} c={m:.3f}  d={_delta:.3f}")
+            s = f"{label}curr={m:.3f}MB delta={_delta:.3f}MB"
+        else:
+            s = f"{label}curr={m:.3f}MB"
+
+        if self.add_peak:
+            s += f" peak={self.peak():.3f}MB"
+
+        if as_str:
+            return s
         else:
-            print(f"{label} c={m:.3f}")
+            print(s)
 
 
 class Diag:
-    def __init__(self, name=""):
+    def __init__(self, name="", peak=False):
         self.time = TimeDiag(name)
-        self.memory = MemoryDiag(name)
+        self.memory = MemoryDiag(name, peak=peak)
 
     def __call__(self, label=""):
-        self.time(label)
-        self.memory(label)
+        if label:
+            label = f"[{label:10}] "
+        return f"{label}{self.time(as_str=True)} {self.memory(as_str=True)}"
+
+    def peak(self):
+        return self.memory.peak()
+
+
+def metadata_cache_diag(fieldlist):
+    r = defaultdict(int)
+    for f in fieldlist:
+        collect_field_metadata_cache_diag(f, r)
+        # try:
+        #     md_cache = f._diag()
+        #     for k in ["metadata_cache_hits", "metadata_cache_misses", "metadata_cache_size"]:
+        #         r[k] += md_cache[k]
+        # except Exception:
+        #     pass
+    return r
+
+
+def collect_field_metadata_cache_diag(field, r):
+    try:
+        md_cache = field_cache_diag(field)
+        for k in ["metadata_cache_hits", "metadata_cache_misses", "metadata_cache_size"]:
+            r[k] += md_cache[k]
+    except Exception:
+        pass
+
+
+def field_cache_diag(field):
+    r = defaultdict(int)
+    try:
+        md_cache = field.metadata()._cache
+        r["metadata_cache_size"] += len(md_cache)
+        r["metadata_cache_hits"] += md_cache.hits
+        r["metadata_cache_misses"] += md_cache.misses
+    except Exception:
+        pass
+    return r


Test patch:
diff --git a/tests/grib/grib_fixtures.py b/tests/grib/grib_fixtures.py
index 95a9794..46ccc6e 100644
--- a/tests/grib/grib_fixtures.py
+++ b/tests/grib/grib_fixtures.py
@@ -9,7 +9,6 @@
 # nor does it submit to any jurisdiction.
 #
 
-
 from earthkit.data import from_source
 from earthkit.data.testing import ARRAY_BACKENDS
 from earthkit.data.testing import earthkit_examples_file
@@ -17,15 +16,15 @@ from earthkit.data.testing import earthkit_test_data_file
 from earthkit.data.utils.array import get_backend
 
 
-def load_array_fieldlist(path, array_backend):
-    ds = from_source("file", path)
+def load_array_fieldlist(path, array_backend, **kwargs):
+    ds = from_source("file", path, **kwargs)
     return ds.to_fieldlist(array_backend=array_backend)
     # return FieldList.from_array(
     #     ds.values, [m.override(generatingProcessIdentifier=120) for m in ds.metadata()]
     # )
 
 
-def load_grib_data(filename, fl_type, folder="example"):
+def load_grib_data(filename, fl_type, folder="example", **kwargs):
     if isinstance(filename, str):
         filename = [filename]
 
@@ -37,10 +36,19 @@ def load_grib_data(filename, fl_type, folder="example"):
         raise ValueError(f"Invalid folder={folder}")
 
     if fl_type == "file":
-        return from_source("file", path), get_backend("numpy")
+        return from_source("file", path, **kwargs), get_backend("numpy")
     elif fl_type in ARRAY_BACKENDS:
         array_backend = fl_type
-        return load_array_fieldlist(path, array_backend), get_backend(array_backend)
+        return load_array_fieldlist(path, array_backend, **kwargs), get_backend(array_backend)
+    elif fl_type == "array":
+        return load_array_fieldlist(path, "numpy", **kwargs), get_backend("numpy")
+    elif fl_type == "memory":
+        assert len(path) == 1
+        with open(path[0], "rb") as f:
+            ds = from_source("stream", f, read_all=True, **kwargs)
+            len(ds)  # force reading
+            return ds, get_backend("numpy")
+
     else:
         raise ValueError(f"Invalid fl_type={fl_type}")
 
diff --git a/tests/grib/test_grib_cache.py b/tests/grib/test_grib_cache.py
index 47edbe7..c86533a 100644
--- a/tests/grib/test_grib_cache.py
+++ b/tests/grib/test_grib_cache.py
@@ -9,13 +9,20 @@
 # nor does it submit to any jurisdiction.
 #
 
+import os
 import pickle
+import sys
 
 import pytest
 
 from earthkit.data import from_source
 from earthkit.data import settings
 from earthkit.data.testing import earthkit_examples_file
+from earthkit.data.utils.diag import field_cache_diag
+
+here = os.path.dirname(__file__)
+sys.path.insert(0, here)
+from grib_fixtures import load_grib_data  # noqa: E402
 
 
 class TestMetadataCache:
@@ -56,7 +63,7 @@ def _check_diag(diag, ref):
 
 @pytest.mark.parametrize("handle_cache_size", [1, 5])
 @pytest.mark.parametrize("serialise", [True, False])
-def test_grib_cache_basic_patched(handle_cache_size, serialise, patch_metadata_cache):
+def test_grib_cache_basic_file_patched(handle_cache_size, serialise, patch_metadata_cache):
 
     with settings.temporary(
         {
@@ -67,6 +74,7 @@ def test_grib_cache_basic_patched(handle_cache_size, serialise, patch_metadata_c
         }
     ):
         ds = from_source("file", earthkit_examples_file("tuv_pl.grib"))
+
         if serialise:
             pickled_f = pickle.dumps(ds)
             ds = pickle.loads(pickled_f)
@@ -79,7 +87,7 @@ def test_grib_cache_basic_patched(handle_cache_size, serialise, patch_metadata_c
         # for f in ds:
         #     print(f.metadata()._cache.data)
 
-        diag = ds._diag()
+        diag = ds._cache_diag()
         ref = {
             "field_cache_size": 18,
             "field_create_count": 18,
@@ -90,13 +98,13 @@ def test_grib_cache_basic_patched(handle_cache_size, serialise, patch_metadata_c
             "metadata_cache_misses": 18 * 6,
             "metadata_cache_size": 18 * 6,
         }
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         for i, f in enumerate(ds):
             assert i in ds._field_manager.cache, f"{i} not in cache"
             assert id(f) == id(ds._field_manager.cache[i]), f"{i} not the same object"
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # unique values repeated
         vals = ds.unique_values("paramId", "levelist", "levtype", "valid_datetime")
@@ -113,11 +121,11 @@ def test_grib_cache_basic_patched(handle_cache_size, serialise, patch_metadata_c
             "metadata_cache_misses": 18 * 6,
             "metadata_cache_size": 18 * 6,
         }
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # order by
         ds.order_by(["levelist", "valid_datetime", "paramId", "levtype"])
-        diag = ds._diag()
+        diag = ds._cache_diag()
         ref = {
             "field_cache_size": 18,
             "field_create_count": 18,
@@ -127,7 +135,7 @@ def test_grib_cache_basic_patched(handle_cache_size, serialise, patch_metadata_c
             "metadata_cache_misses": 18 * 6,
             "metadata_cache_size": 18 * 6,
         }
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         assert diag["metadata_cache_hits"] >= 18 * 4
 
@@ -137,7 +145,7 @@ def test_grib_cache_basic_patched(handle_cache_size, serialise, patch_metadata_c
         assert ds[0].handle == md._handle
 
 
-def test_grib_cache_basic_non_patched():
+def test_grib_cache_basic_file_non_patched():
     """This test is the same as test_grib_cache_basic but without the patch_metadata_cache fixture.
     So metadata cache hits and misses are not counted."""
 
@@ -165,13 +173,13 @@ def test_grib_cache_basic_non_patched():
             # "metadata_cache_misses": 18 * 6,
             "metadata_cache_size": 18 * 6,
         }
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         for i, f in enumerate(ds):
             assert i in ds._field_manager.cache, f"{i} not in cache"
             assert id(f) == id(ds._field_manager.cache[i]), f"{i} not the same object"
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # unique values repeated
         vals = ds.unique_values("paramId", "levelist", "levtype", "valid_datetime")
@@ -188,7 +196,7 @@ def test_grib_cache_basic_non_patched():
             # "metadata_cache_misses": 18 * 6,
             "metadata_cache_size": 18 * 6,
         }
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # order by
         ds.order_by(["levelist", "valid_datetime", "paramId", "levtype"])
@@ -201,7 +209,7 @@ def test_grib_cache_basic_non_patched():
             # "metadata_cache_misses": 18 * 6,
             "metadata_cache_size": 18 * 6,
         }
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # metadata object is not decoupled from the field object
         md = ds[0].metadata()
@@ -209,6 +217,75 @@ def test_grib_cache_basic_non_patched():
         assert ds[0].handle == md._handle
 
 
+@pytest.mark.parametrize("serialise", [True, False])
+@pytest.mark.parametrize("fl_type", ["file", "array", "memory"])
+def test_grib_cache_basic_metadata_patched(serialise, fl_type, patch_metadata_cache):
+
+    with settings.temporary(
+        {
+            "grib-field-policy": "persistent",
+            "grib-handle-policy": "cache",
+            "grib-handle-cache-size": 1,
+            "use-grib-metadata-cache": True,
+        }
+    ):
+        ds, _ = load_grib_data("tuv_pl.grib", fl_type)
+
+        if serialise:
+            pickled_f = pickle.dumps(ds)
+            ds = pickle.loads(pickled_f)
+
+        assert len(ds) == 18
+
+        # unique values
+        ref_vals = ds.unique_values("paramId", "levelist", "levtype", "valid_datetime")
+
+        # for f in ds:
+        #     print(f.metadata()._cache.data)
+
+        diag = ds._cache_diag()
+        ref = {
+            "metadata_cache_hits": 0,
+            "metadata_cache_misses": 18 * 6,
+            "metadata_cache_size": 18 * 6,
+        }
+        _check_diag(ds._cache_diag(), ref)
+
+        # unique values repeated
+        vals = ds.unique_values("paramId", "levelist", "levtype", "valid_datetime")
+
+        assert vals == ref_vals
+
+        ref = {
+            "metadata_cache_hits": 18 * 4,
+            "metadata_cache_misses": 18 * 6,
+            "metadata_cache_size": 18 * 6,
+        }
+        _check_diag(ds._cache_diag(), ref)
+
+        # order by
+        ds.order_by(["levelist", "valid_datetime", "paramId", "levtype"])
+        diag = ds._cache_diag()
+        ref = {
+            "metadata_cache_misses": 18 * 6,
+            "metadata_cache_size": 18 * 6,
+        }
+        _check_diag(ds._cache_diag(), ref)
+
+        assert diag["metadata_cache_hits"] >= 18 * 4
+
+        # metadata object is not decoupled from the field object
+        md = ds[0].metadata()
+        if fl_type != "array":
+            # handle is taken from the field
+            assert hasattr(md, "_field")
+            assert ds[0].handle == md._handle
+        else:
+            # handle is not taken from the metadata
+            assert not hasattr(md, "_field")
+            assert ds[0].handle == md._handle
+
+
 def test_grib_cache_options_1(patch_metadata_cache):
     with settings.temporary(
         {
@@ -237,13 +314,13 @@ def test_grib_cache_options_1(patch_metadata_cache):
             "metadata_cache_size": 18 * 6,
         }
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         for i, f in enumerate(ds):
             assert i in ds._field_manager.cache, f"{i} not in cache"
             assert id(f) == id(ds._field_manager.cache[i]), f"{i} not the same object"
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # metadata object is not decoupled from the field object
         md = ds[0].metadata()
@@ -257,33 +334,36 @@ def test_grib_cache_options_1(patch_metadata_cache):
         assert first._handle is None
 
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 6, "metadata_cache_size": 6}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 6, "metadata_cache_size": 6},
         )
 
         # key already cached
         first.metadata("levelist", default=None)
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 1, "metadata_cache_misses": 6, "metadata_cache_size": 6}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 1, "metadata_cache_misses": 6, "metadata_cache_size": 6},
         )
 
         ref["metadata_cache_hits"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # uncached key
         first.metadata("level")
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 1, "metadata_cache_misses": 7, "metadata_cache_size": 7}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 1, "metadata_cache_misses": 7, "metadata_cache_size": 7},
         )
 
         ref["handle_create_count"] += 1
         ref["metadata_cache_misses"] += 1
         ref["metadata_cache_size"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         assert first.handle != md._handle
 
         ref["handle_create_count"] += 2
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
 
 def test_grib_cache_options_2(patch_metadata_cache):
@@ -314,13 +394,13 @@ def test_grib_cache_options_2(patch_metadata_cache):
             "metadata_cache_size": 18 * 6,
         }
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         for i, f in enumerate(ds):
             assert i in ds._field_manager.cache, f"{i} not in cache"
             assert id(f) == id(ds._field_manager.cache[i]), f"{i} not the same object"
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # metadata object is not decoupled from the field object
         md = ds[0].metadata()
@@ -335,34 +415,37 @@ def test_grib_cache_options_2(patch_metadata_cache):
         assert first._handle == md._handle
         assert first.handle == first._handle
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 6, "metadata_cache_size": 6}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 6, "metadata_cache_size": 6},
         )
 
         # key already cached
         first.metadata("levelist", default=None)
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 1, "metadata_cache_misses": 6, "metadata_cache_size": 6}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 1, "metadata_cache_misses": 6, "metadata_cache_size": 6},
         )
 
         ref["metadata_cache_hits"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # uncached key
         first.metadata("level")
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 1, "metadata_cache_misses": 7, "metadata_cache_size": 7}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 1, "metadata_cache_misses": 7, "metadata_cache_size": 7},
         )
 
         ref["metadata_cache_misses"] += 1
         ref["metadata_cache_size"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         assert first.handle == md._handle
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
 
 def test_grib_cache_options_3(patch_metadata_cache):
@@ -394,13 +477,13 @@ def test_grib_cache_options_3(patch_metadata_cache):
             "metadata_cache_size": 18 * 6,
         }
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         for i, f in enumerate(ds):
             assert i in ds._field_manager.cache, f"{i} not in cache"
             assert id(f) == id(ds._field_manager.cache[i]), f"{i} not the same object"
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # metadata object is not decoupled from the field object
         md = ds[0].metadata()
@@ -414,32 +497,35 @@ def test_grib_cache_options_3(patch_metadata_cache):
         assert first._handle is None
 
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 6, "metadata_cache_size": 6}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 6, "metadata_cache_size": 6},
         )
 
         # key already cached
         first.metadata("levelist", default=None)
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 1, "metadata_cache_misses": 6, "metadata_cache_size": 6}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 1, "metadata_cache_misses": 6, "metadata_cache_size": 6},
         )
 
         ref["metadata_cache_hits"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # uncached key
         first.metadata("level")
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 1, "metadata_cache_misses": 7, "metadata_cache_size": 7}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 1, "metadata_cache_misses": 7, "metadata_cache_size": 7},
         )
 
         ref["handle_create_count"] += 1
         ref["metadata_cache_misses"] += 1
         ref["metadata_cache_size"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         assert first.handle == md._handle
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
 
 def test_grib_cache_options_4(patch_metadata_cache):
@@ -468,7 +554,7 @@ def test_grib_cache_options_4(patch_metadata_cache):
             "metadata_cache_size": 0,
         }
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         assert ds._field_manager.cache is None
         assert ds._handle_manager.cache is None
@@ -479,7 +565,7 @@ def test_grib_cache_options_4(patch_metadata_cache):
         assert ds[0]._handle != md._handle
         ref["field_create_count"] += 2
         ref["handle_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # keep a reference to the field
         first = ds[0]
@@ -487,50 +573,56 @@ def test_grib_cache_options_4(patch_metadata_cache):
         assert md._field == first
         assert first._handle is None
         ref["field_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
 
         first.metadata("levelist", default=None)
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 1, "metadata_cache_size": 1}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 1, "metadata_cache_size": 1},
         )
 
         first.metadata("levelist", default=None)
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 1, "metadata_cache_misses": 1, "metadata_cache_size": 1}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 1, "metadata_cache_misses": 1, "metadata_cache_size": 1},
         )
 
         ref["handle_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # repeat with indexed field
         _check_diag(
-            ds[0]._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(ds[0]),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
 
         ref["field_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         ds[0].metadata("levelist", default=None)
         _check_diag(
-            ds[0]._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(ds[0]),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
 
         ref["field_create_count"] += 2
         ref["handle_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         ds[0].metadata("levelist", default=None)
         _check_diag(
-            ds[0]._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(ds[0]),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
 
         ref["field_create_count"] += 2
         ref["handle_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
 
 def test_grib_cache_options_5(patch_metadata_cache):
@@ -559,7 +651,7 @@ def test_grib_cache_options_5(patch_metadata_cache):
             "metadata_cache_size": 0,
         }
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         assert ds._field_manager.cache is None
         assert ds._handle_manager.cache is None
@@ -570,7 +662,7 @@ def test_grib_cache_options_5(patch_metadata_cache):
         assert ds[0]._handle != md._handle
         ref["field_create_count"] += 2
         ref["handle_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # keep a reference to the field
         first = ds[0]
@@ -578,52 +670,58 @@ def test_grib_cache_options_5(patch_metadata_cache):
         assert md._field == first
         assert first._handle is None
         ref["field_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
 
         first.metadata("levelist", default=None)
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 1, "metadata_cache_size": 1}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 1, "metadata_cache_size": 1},
         )
 
         assert first._handle is not None
 
         first.metadata("levelist", default=None)
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 1, "metadata_cache_misses": 1, "metadata_cache_size": 1}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 1, "metadata_cache_misses": 1, "metadata_cache_size": 1},
         )
 
         assert first._handle is not None
 
         ref["handle_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # repeat with indexed field
         _check_diag(
-            ds[0]._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(ds[0]),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
 
         ref["field_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         ds[0].metadata("levelist", default=None)
         _check_diag(
-            ds[0]._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(ds[0]),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
         ref["field_create_count"] += 2
         ref["handle_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         ds[0].metadata("levelist", default=None)
         _check_diag(
-            ds[0]._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(ds[0]),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
         ref["field_create_count"] += 2
         ref["handle_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
 
 def test_grib_cache_options_6(patch_metadata_cache):
@@ -652,7 +750,7 @@ def test_grib_cache_options_6(patch_metadata_cache):
             "metadata_cache_size": 0,
         }
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         assert ds._field_manager.cache is None
         assert ds._handle_manager.cache is not None
@@ -663,7 +761,7 @@ def test_grib_cache_options_6(patch_metadata_cache):
         assert ds[0]._handle != md._handle
         ref["field_create_count"] += 2
         ref["handle_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # keep a reference to the field
         first = ds[0]
@@ -671,48 +769,54 @@ def test_grib_cache_options_6(patch_metadata_cache):
         assert md._field == first
         assert first._handle is None
         ref["field_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
 
         first.metadata("levelist", default=None)
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 1, "metadata_cache_size": 1}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 1, "metadata_cache_size": 1},
         )
 
         first.metadata("levelist", default=None)
         _check_diag(
-            first._diag(), {"metadata_cache_hits": 1, "metadata_cache_misses": 1, "metadata_cache_size": 1}
+            field_cache_diag(first),
+            {"metadata_cache_hits": 1, "metadata_cache_misses": 1, "metadata_cache_size": 1},
         )
 
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         # repeat with indexed field
         _check_diag(
-            ds[0]._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(ds[0]),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
 
         ref["field_create_count"] += 1
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         ds[0].metadata("levelist", default=None)
         _check_diag(
-            ds[0]._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(ds[0]),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
         ref["field_create_count"] += 2
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
         ds[0].metadata("levelist", default=None)
         _check_diag(
-            ds[0]._diag(), {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0}
+            field_cache_diag(ds[0]),
+            {"metadata_cache_hits": 0, "metadata_cache_misses": 0, "metadata_cache_size": 0},
         )
         ref["field_create_count"] += 2
-        _check_diag(ds._diag(), ref)
+        _check_diag(ds._cache_diag(), ref)
 
 
-def test_grib_cache_use_kwargs_1():
+def test_grib_cache_file_use_kwargs_1():
     _kwargs = {
         "grib_field_policy": "temporary",
         "grib_handle_policy": "persistent",
@@ -737,10 +841,10 @@ def test_grib_cache_use_kwargs_1():
         "metadata_cache_size": 0,
     }
 
-    _check_diag(ds._diag(), ref)
+    _check_diag(ds._cache_diag(), ref)
 
 
-def test_grib_cache_use_kwargs_2():
+def test_grib_cache_file_use_kwargs_2():
     _kwargs = {
         "grib-field-policy": "temporary",
         "grib_handle_policy": "persistent",
@@ -750,3 +854,87 @@ def test_grib_cache_use_kwargs_2():
 
     with pytest.raises(KeyError):
         from_source("file", earthkit_examples_file("tuv_pl.grib"), **_kwargs)
+
+
+@pytest.mark.parametrize("fl_type", ["file", "array", "memory"])
+def test_grib_cache_metadata_use_kwargs_1(fl_type, patch_metadata_cache):
+    with settings.temporary(
+        {
+            "grib-field-policy": "persistent",
+            "grib-handle-policy": "cache",
+            "grib-handle-cache-size": 1,
+            "use-grib-metadata-cache": False,
+        }
+    ):
+
+        _kwargs = {
+            "use_grib_metadata_cache": True,
+        }
+
+        ds, _ = load_grib_data("tuv_pl.grib", fl_type, **_kwargs)
+
+        assert len(ds) == 18
+
+        # unique values
+        ds.unique_values("paramId", "levelist", "levtype", "valid_datetime")
+
+        ref = {
+            "metadata_cache_hits": 0,
+            "metadata_cache_misses": 108,
+            "metadata_cache_size": 108,
+        }
+
+        _check_diag(ds._cache_diag(), ref)
+
+        # unique values
+        ds.unique_values("paramId", "levelist", "levtype", "valid_datetime")
+
+        ref = {
+            "metadata_cache_hits": 72,
+            "metadata_cache_misses": 108,
+            "metadata_cache_size": 108,
+        }
+
+        _check_diag(ds._cache_diag(), ref)
+
+
+@pytest.mark.parametrize("fl_type", ["file", "array", "memory"])
+def test_grib_cache_metadata_use_kwargs_2(fl_type, patch_metadata_cache):
+    with settings.temporary(
+        {
+            "grib-field-policy": "persistent",
+            "grib-handle-policy": "cache",
+            "grib-handle-cache-size": 1,
+            "use-grib-metadata-cache": True,
+        }
+    ):
+
+        _kwargs = {
+            "use_grib_metadata_cache": False,
+        }
+
+        ds, _ = load_grib_data("tuv_pl.grib", fl_type, **_kwargs)
+
+        assert len(ds) == 18
+
+        # unique values
+        ds.unique_values("paramId", "levelist", "levtype", "valid_datetime")
+
+        ref = {
+            "metadata_cache_hits": 0,
+            "metadata_cache_misses": 0,
+            "metadata_cache_size": 0,
+        }
+
+        _check_diag(ds._cache_diag(), ref)
+
+        # unique values
+        ds.unique_values("paramId", "levelist", "levtype", "valid_datetime")
+
+        ref = {
+            "metadata_cache_hits": 0,
+            "metadata_cache_misses": 0,
+            "metadata_cache_size": 0,
+        }
+
+        _check_diag(ds._cache_diag(), ref)