Repository: ecmwf/earthkit-data
Pull request #430 (2024-08-09 19:20:47)

Implement standalone grib metadata
This PR adds the following changes:

- `GribMetadata` becomes abstract with two direct subclasses: `GribFieldMetadata` and  `StandAloneGribMetadata` 
- `GribField.metadata()` returns a `GribFieldMetadata` object. This does not possess a GRIB handle but accesses it via the field it belongs to
- `GribMetadata.override()` returns a `StandAloneGribMetadata` object, which possesses it owns GRIB handle


codecov-commenter: ## [Codecov](https://app.codecov.io/gh/ecmwf/earthkit-data/pull/430?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 90.16%. Comparing base [(`b22e2a1`)](https://app.codecov.io/gh/ecmwf/earthkit-data/commit/b22e2a1071ec778dc49064bbbde8094dbb641360?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf) to head [(`90b1319`)](https://app.codecov.io/gh/ecmwf/earthkit-data/commit/90b13198f61f2072ea6bacdc4241b81c4c126915?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf).

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           develop     #430      +/-   ##
===========================================
+ Coverage    90.11%   90.16%   +0.04%     
===========================================
  Files          122      122              
  Lines         7955     7995      +40     
  Branches       680      682       +2     
===========================================
+ Hits          7169     7209      +40     
  Misses         655      655              
  Partials       131      131              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/ecmwf/earthkit-data/pull/430?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf).


Patch:
diff --git a/docs/api.rst b/docs/api.rst
index d5f064d..aabd383 100644
--- a/docs/api.rst
+++ b/docs/api.rst
@@ -18,6 +18,8 @@ Metadata
 - :py:class:`~data.core.metadata.Metadata`
 - :py:class:`~data.core.metadata.RawMetadata`
 - :py:class:`~data.readers.grib.metadata.GribMetadata`
+- :py:class:`~data.readers.grib.metadata.GribFieldMetadata`
+- :py:class:`~data.readers.grib.metadata.StandAloneGribMetadata`
 
 Numpy fields
 ---------------
diff --git a/docs/examples/grib_metadata_object.ipynb b/docs/examples/grib_metadata_object.ipynb
new file mode 100644
index 0000000..21fe816
--- /dev/null
+++ b/docs/examples/grib_metadata_object.ipynb
@@ -0,0 +1,786 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "d08d9138-0a0a-495c-b2f0-27b47e45847c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "## GRIB: using the metadata object"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "25dd1879-9f88-4555-a847-a0ed45e3d6e4",
+   "metadata": {
+    "editable": true,
+    "raw_mimetype": "text/restructuredtext",
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "We will work with a GRIB file containing 6 messages. First we ensure the example file is available, then read the file with :ref:`from_source() <data-sources-file>`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "d1294cff-422a-4ed5-a1cf-87c6cf9195ad",
+   "metadata": {
+    "editable": true,
+    "raw_mimetype": "",
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import earthkit.data\n",
+    "earthkit.data.download_example_file(\"test6.grib\")\n",
+    "ds = earthkit.data.from_source(\"file\", \"test6.grib\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d3392799-8f51-47f3-95b1-4a53d67a2c29",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "#### The metadata object"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "b2c90171-2ce4-445c-9993-3354286ba570",
+   "metadata": {
+    "editable": true,
+    "raw_mimetype": "text/restructuredtext",
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [],
+    "vscode": {
+     "languageId": "raw"
+    }
+   },
+   "source": [
+    "The recommended way to get field metadata for a given key or keys is to use :py:meth:`~data.readers.grib.codes.GribField.metadata`. See :ref:`/examples/grib_metadata.ipynb` for more examples."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "9daeae60-e906-4bd0-9055-e940218bb12c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'t'"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ds[0].metadata(\"shortName\")"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "9282df0e-2430-44fb-9025-29691bb3a24b",
+   "metadata": {
+    "editable": true,
+    "raw_mimetype": "text/restructuredtext",
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "Internally the field metadata is represented by a GribFieldMetadata object. Calling :py:meth:`~data.readers.grib.codes.GribField.metadata` without any arguments will return this object."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "1b795c50-f942-4417-b7e3-bd4aee4d11f5",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<earthkit.data.readers.grib.metadata.GribFieldMetadata at 0x106c2e320>"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "md = ds[0].metadata()\n",
+    "md"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0ad46cbe-9413-435a-bf5c-db2cb8cf8203",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "It can be used as a dict."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "5d5a9ad8-981e-440e-8327-9b21468ae42c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "('t', 1000)"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "md[\"shortName\"], md.get(\"level\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3d422ff7-c5ef-4f8d-b633-ecd93e418f3f",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "<div class=\"alert alert-warning\">\n",
+    "<b>Warning:</b> this object <b>does not own a GRIB handle</b> but contains a reference to the field, which provides access to the GRIB handle. Therefore if you need to store it for later use it is recommended to create a copy with override() (see below) to decouple it from the field object.</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e50bd099-336f-4952-941b-aa58e93f4c70",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "#### Creating a copy of the metadata object"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "6cd89c6c-bcd5-4f83-b83b-03c28447ae41",
+   "metadata": {
+    "editable": true,
+    "raw_mimetype": "text/restructuredtext",
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "We can create a copy with :py:meth:`~data.readers.grib.metadata.GribMetadata.override`, which always returns a new metadata object storing a cloned GRIB handle, thus it is decoupled from the field. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "1369ca99-40d2-4352-8c9e-94666e5ebeec",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<earthkit.data.readers.grib.metadata.StandAloneGribMetadata at 0x2839bfc70>"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "md_copy = md.override()\n",
+    "md_copy"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "388d83a5-8165-4b3d-a121-2c7f2cb578e1",
+   "metadata": {
+    "editable": true,
+    "raw_mimetype": "text/restructuredtext",
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [],
+    "vscode": {
+     "languageId": "raw"
+    }
+   },
+   "source": [
+    "By default :py:meth:`~data.readers.grib.metadata.GribMetadata.override` is called with ``headers_only_clone=True`` generating the new handle with all the data values (and some related information) removed. With this the resulting object can be significantly smaller, especially if the data section is large. The downside is that now the value related keys either cannot be accessed or give back wrong values. E.g when using the \"average\" key we get:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "568fe5c9-df84-4d49-aee1-ada0e6a15c28",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(279.70703560965404, 47485.4296875)"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "md[\"average\"], md_copy[\"average\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8796976f-a02d-4101-8cff-c14a7730d82c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "To get a copy without shrinking the GRIB handle use ``headers_only_clone=False``."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "ba989ad8-e034-4168-bc29-1d73877edad2",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "md_copy_full = md.override(headers_only_clone=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "15d723c7-2f73-4c14-ab87-2a94de2379f8",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "279.70703560965404"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "md_copy_full[\"average\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3240cef2-baa9-4a87-a83a-dafa97b78e43",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "#### Changing the metadata"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "449f7fe0-222b-456f-bb15-dbe263774f4b",
+   "metadata": {
+    "editable": true,
+    "raw_mimetype": "text/restructuredtext",
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "To change metadata values pass key value pairs or a dict to :py:meth:`~data.readers.grib.metadata.GribMetadata.override`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "40c6d232-03de-402b-82bf-8647e8a7bece",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "('z', 850)"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "md_mod = md.override(shortName=\"z\", level=\"850\")\n",
+    "md_mod[\"shortName\"], md_mod[\"level\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "31b017d2-843a-4dc5-8d49-b8c05f30ed37",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "Since md_mod contains a clone of the GRIB handle the original metadata did not change."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "ef78a3ec-4ea2-4ff5-8c90-e60b5e07e77f",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "('t', 1000)"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "md[\"shortName\"], md[\"level\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "759e4311-2419-4790-883a-84884190a8c7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "#### Creating array fields from metadata and values"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0316233b-8da2-49a0-b8b8-4cedd6aba6b0",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "GRIB metadata objects play a part in building new fieldlist from (altered) values and metadata."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a691f6ef-4b5b-47d0-accf-5e956702b47c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "The following example computes the wind speed on 1000 hPa and creates a new fieldlist with a single field containing the new values. The metadata is taken from the u field and the shortName is modified."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "cb59ad5f-c48b-4943-984d-3abdf48fda8d",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from earthkit.data import FieldList\n",
+    "import numpy as np\n",
+    "u = ds.sel(param=\"u\", level=1000)[0]\n",
+    "v = ds.sel(param=\"v\", level=1000)[0]\n",
+    "speed = np.sqrt(u.values**2 + v.values**2)\n",
+    "md_speed = u.metadata().override(shortName=\"ws\")\n",
+    "ds_speed = FieldList.from_array(speed, md_speed) "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "85c32bfb-c929-404f-add9-9adae40418d2",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>centre</th>\n",
+       "      <th>shortName</th>\n",
+       "      <th>typeOfLevel</th>\n",
+       "      <th>level</th>\n",
+       "      <th>dataDate</th>\n",
+       "      <th>dataTime</th>\n",
+       "      <th>stepRange</th>\n",
+       "      <th>dataType</th>\n",
+       "      <th>number</th>\n",
+       "      <th>gridType</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>ecmf</td>\n",
+       "      <td>ws</td>\n",
+       "      <td>isobaricInhPa</td>\n",
+       "      <td>1000</td>\n",
+       "      <td>20180801</td>\n",
+       "      <td>1200</td>\n",
+       "      <td>0</td>\n",
+       "      <td>an</td>\n",
+       "      <td>0</td>\n",
+       "      <td>regular_ll</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "  centre shortName    typeOfLevel  level  dataDate  dataTime stepRange  \\\n",
+       "0   ecmf        ws  isobaricInhPa   1000  20180801      1200         0   \n",
+       "\n",
+       "  dataType  number    gridType  \n",
+       "0       an       0  regular_ll  "
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ds_speed.ls()"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "a5b3c420-f71d-4a58-8cbe-761a0ce73a9b",
+   "metadata": {
+    "editable": true,
+    "raw_mimetype": "text/restructuredtext",
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [],
+    "vscode": {
+     "languageId": "raw"
+    }
+   },
+   "source": [
+    "Please note that the resulting :py:class:`~data.sources.array_list.ArrayFieldList` always contains a :py:class:`~data.readers.grib.metadata.RestrictedGribMetadata` object for each field. These objects possess their own GRIB handles, which is ensured by creating a copy with ``override(headers_only_clone=True)`` when needed. On top of that metadata access is limited to keys not related to data values. Getting metadata on any other keys will throw an exception. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "c6fe87ed-ee88-4f4d-a2b6-9401b364e2df",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<earthkit.data.readers.grib.metadata.RestrictedGribMetadata at 0x2839fd2a0>"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ds_speed[0].metadata()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "27686ac4-9382-4916-ad0e-be96a649d034",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'Wind speed'"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ds_speed[0].metadata(\"name\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "dc28fa77-4020-431f-ad37-e480a69f9d7f",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "KeyError 'average'\n"
+     ]
+    }
+   ],
+   "source": [
+    "try:\n",
+    "    ds_speed[0].metadata(\"average\")\n",
+    "except KeyError as e:\n",
+    "    print(f\"KeyError {e}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "331fa5aa-a134-423e-a4e3-bc8adf779297",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "However, strictly speaking these keys do not represent metadata and they can be easily computed from the field values when needed."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "8eab3462-3661-4fc1-9d23-8be05dc99cd8",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "7.450183054360252"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ds_speed[0].values.mean()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "351236ef-3b2a-4d1e-bef9-3a7154af2270",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "dev_ecc",
+   "language": "python",
+   "name": "dev_ecc"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/examples/index.rst b/docs/examples/index.rst
index f7883af..7d992f1 100644
--- a/docs/examples/index.rst
+++ b/docs/examples/index.rst
@@ -40,6 +40,7 @@ GRIB
     grib_selection.ipynb
     grib_indexing.ipynb
     grib_missing.ipynb
+    grib_metadata_object.ipynb
     numpy_fieldlist.ipynb
     grib_array_backends.ipynb
     grib_nearest_gridpoint.ipynb
diff --git a/docs/skip_api_rules.py b/docs/skip_api_rules.py
index 8e247d0..9eb514c 100644
--- a/docs/skip_api_rules.py
+++ b/docs/skip_api_rules.py
@@ -103,6 +103,26 @@ _skip_methods = {
         "statistics",
         "xarray_open_dataset_kwargs",
     ],
+    "data.source.numpy_list.ArrayField": ["merge", "mutate"],
+    "data.source.numpy_list.ArrayFieldList": [
+        "cache_file",
+        "dataset",
+        "from_dict",
+        "from_mask",
+        "from_multi",
+        "from_slice",
+        "full",
+        "graph",
+        "ignore",
+        "merge",
+        "mutate",
+        "new_mask_index",
+        "parent",
+        "scaled",
+        "settings",
+        "statistics",
+        "xarray_open_dataset_kwargs",
+    ],
 }
 
 
@@ -120,8 +140,10 @@ def _skip_api_items(app, what, name, obj, skip, options):
         "data.readers.bufr.bufr",
         "data.readers.grib.codes",
         "data.readers.grib.index",
+        "data.readers.grib.metadata",
         "data.readers.csv",
         "data.sources",
+        "data.sources.array_list",
         "data.sources.numpy_list",
         "data.utils",
         "data.utils.bbox",
@@ -136,9 +158,11 @@ def _skip_api_items(app, what, name, obj, skip, options):
         "data.readers.bufr.bufr",
         "data.readers.grib",
         "data.readers.grib.index",
+        "data.readers.grib.metadata",
         "data.readers.csv",
         "data.sources",
         "data.sources.numpy_list",
+        "data.sources.numpy_list",
         "data.utils",
         "data.utils.bbox",
     ]:
@@ -154,9 +178,14 @@ def _skip_api_items(app, what, name, obj, skip, options):
         "data.readers.grib.codes.GribField",
         "data.readers.grib.index.GribFieldList",
         "data.readers.grib.metadata.GribMetadata",
+        "data.readers.grib.metadata.GribFieldMetadata",
+        "data.readers.grib.metadata.StandAloneGribMetadata",
+        "data.readers.grib.metadata.RestrictedGribMetadata",
         "data.readers.csv.CSVReader",
         "data.sources.numpy_list.NumpyField",
         "data.sources.numpy_list.NumpyFieldList",
+        "data.sources.array_list.ArrayField",
+        "data.sources.array_list.ArrayFieldList",
         "data.utils.bbox.BoundingBox",
     ]:
         skip = True
diff --git a/src/earthkit/data/core/fieldlist.py b/src/earthkit/data/core/fieldlist.py
index f49cca5..f899f51 100644
--- a/src/earthkit/data/core/fieldlist.py
+++ b/src/earthkit/data/core/fieldlist.py
@@ -115,14 +115,11 @@ class Field(Base):
             return self._array_backend.array_ns.reshape(v, n)
         return v
 
-    def _make_metadata(self):
-        r"""Create a field metadata object."""
-        self._not_implemented()
-
     @property
     def _metadata(self):
         r"""Metadata: Get the object representing the field's metadata."""
         if self.__metadata is None:
+            # TODO: remove this legacy method
             self.__metadata = self._make_metadata()
         return self.__metadata
 
@@ -464,7 +461,8 @@ class Field(Base):
     def metadata(self, *keys, astype=None, **kwargs):
         r"""Return metadata values from the field.
 
-        When called without any arguments returns a :obj:`Metadata` object.
+        When called without any arguments returns a :obj:`Metadata` object, which for GRIB data
+        it always owns its own copy of the ecCodes handle of the GRIB message.
 
         Parameters
         ----------
diff --git a/src/earthkit/data/core/metadata.py b/src/earthkit/data/core/metadata.py
index ef8b758..4a36fdd 100644
--- a/src/earthkit/data/core/metadata.py
+++ b/src/earthkit/data/core/metadata.py
@@ -37,6 +37,10 @@ class Metadata(metaclass=ABCMeta):
 
     extra = None
 
+    def __init__(self, extra=None):
+        if extra is not None:
+            self.extra = extra
+
     def __iter__(self):
         """Return an iterator over the metadata keys."""
         return iter(self.keys())
diff --git a/src/earthkit/data/readers/grib/codes.py b/src/earthkit/data/readers/grib/codes.py
index b10163d..9041d07 100644
--- a/src/earthkit/data/readers/grib/codes.py
+++ b/src/earthkit/data/readers/grib/codes.py
@@ -9,12 +9,13 @@
 
 import logging
 import os
+from functools import cached_property
 
 import eccodes
 import numpy as np
 
 from earthkit.data.core.fieldlist import Field
-from earthkit.data.readers.grib.metadata import GribMetadata
+from earthkit.data.readers.grib.metadata import GribFieldMetadata
 from earthkit.data.utils.message import CodesHandle
 from earthkit.data.utils.message import CodesMessagePositionIndex
 from earthkit.data.utils.message import CodesReader
@@ -266,8 +267,9 @@ class GribField(Field):
             self._offset = int(self.handle.get("offset"))
         return self._offset
 
-    def _make_metadata(self):
-        return GribMetadata(self.handle)
+    @cached_property
+    def _metadata(self):
+        return GribFieldMetadata(self)
 
     def __repr__(self):
         return "GribField(%s,%s,%s,%s,%s,%s)" % (
diff --git a/src/earthkit/data/readers/grib/metadata.py b/src/earthkit/data/readers/grib/metadata.py
index 03028c5..500d8c0 100644
--- a/src/earthkit/data/readers/grib/metadata.py
+++ b/src/earthkit/data/readers/grib/metadata.py
@@ -9,6 +9,7 @@
 
 import datetime
 import warnings
+from abc import abstractmethod
 from functools import cached_property
 
 from earthkit.data.core.geography import Geography
@@ -239,26 +240,11 @@ class GribFieldGeography(Geography):
 
 
 class GribMetadata(Metadata):
-    """Represent the metadata of a GRIB field.
-
-    Parameters
-    ----------
-    handle: :obj:`GribCodesHandle`
-        Object representing the ecCodes GRIB handle of the field.
-
-    :obj:`GribMetadata` is created internally by a :obj:`GribField` and we can use
-    the field's :meth:`metadata` method to access it.
-
-    >>> ds = earthkit.data.from_source("file", "docs/examples/test4.grib")
-    >>> md = ds[0].metadata()
-    >>> md["shortName"]
-    't'
-    >>> md.get("shortName")
-    't'
-    >>> md.get("nonExistentKey")
-    >>> md.get("nonExistentKey", 12)
-    12
+    """GRIB metadata.
 
+    :obj:`GribMetadata` is an abstract class and should not be instantiated directly.
+    There are two concrete implementations: :class:`GribFieldMetadata` and
+    :class:`StandAloneGribMetadata`.
     """
 
     LS_KEYS = [
@@ -306,13 +292,14 @@ class GribMetadata(Metadata):
 
     __handle_type = None
 
-    def __init__(self, handle, extra=None):
-        if not isinstance(handle, self._handle_type()):
-            raise TypeError(f"GribMetadata: expected handle type {self._handle_type()}, got {type(handle)}")
-        self._handle = handle
+    @abstractmethod
+    def _handle(self):
+        pass
+
+    def __init__(self, shrunk=False, **kwargs):
         self._geo = None
-        if extra is not None:
-            self.extra = extra
+        self._shrunk = shrunk
+        super().__init__(**kwargs)
 
     @staticmethod
     def _handle_type():
@@ -360,7 +347,7 @@ class GribMetadata(Metadata):
     def _is_custom_key(self, key):
         return key in self.CUSTOM_KEYS
 
-    def override(self, *args, **kwargs):
+    def override(self, *args, headers_only_clone=True, **kwargs):
         d = dict(*args, **kwargs)
 
         new_value_size = None
@@ -373,17 +360,27 @@ class GribMetadata(Metadata):
             md, new_value_size = GridSpecConverter.to_metadata(gridspec, edition=edition)
             d.update(md)
 
-        handle = self._handle.clone(headers_only=True)
-        # whether headers_only=True works depends on the eccCodes version and the
-        # message properties. We check it by comparing the message lengths.
-        shrunk = handle.get_long("totalLength") < self._handle.get_long("totalLength")
-
-        # some keys, needed later, are not copied into the clone when
-        # headers_only=True. We store them as extra keys.
-        if shrunk:
-            extra = {"bitsPerValue": self._handle.get("bitsPerValue", default=0)}
+        handle = self._handle.clone(headers_only=headers_only_clone)
+
+        if self._shrunk:
+            shrunk = True
+            extra = self.extra
+        elif headers_only_clone:
+            # whether headers_only=True works depends on the eccCodes version and the
+            # message properties. We check it by comparing the message lengths.
+            shrunk = handle.get_long("totalLength") < self._handle.get_long("totalLength")
+
+            # some keys, needed later, are not copied into the clone when
+            # headers_only=True. We store them as extra keys.
+            if shrunk:
+                extra = {"bitsPerValue": self._handle.get("bitsPerValue", default=0)}
+            else:
+                extra = None
+        else:
+            shrunk = False
 
-        handle.set_multiple(d)
+        if d:
+            handle.set_multiple(d)
 
         # we need to set the values to the new size otherwise the clone generated
         # with headers_only=True will be inconsistent
@@ -393,7 +390,7 @@ class GribMetadata(Metadata):
             vals = np.zeros(new_value_size)
             handle.set_values(vals)
 
-        return GribMetadata(handle, extra=extra)
+        return StandAloneGribMetadata(handle, extra=extra, shrunk=shrunk)
 
     def as_namespace(self, namespace=None):
         r"""Return all the keys/values from a namespace.
@@ -491,13 +488,75 @@ class GribMetadata(Metadata):
 
         return format_namespace_dump(r, selected="parameter", details=self.__class__.__name__, **kwargs)
 
+
+class GribFieldMetadata(GribMetadata):
+    """Represent the metadata of a GRIB field.
+
+    :obj:`GribFieldMetadata` is created internally by a :obj:`GribField`. It does not
+    own the ecCodes GRIB handle but can access it through the :obj:`GribField`.
+    Calling :meth:`metadata` without arguments on a :obj:`GribField` returns this object.
+    """
+
+    def __init__(self, field, **kwargs):
+        self._field = field
+        assert field is not None
+        super().__init__(**kwargs)
+
+    @property
+    def _handle(self):
+        return self._field.handle
+
+    def _hide_internal_keys(self):
+        r = self.override()
+        return RestrictedGribMetadata(r)
+
+
+class StandAloneGribMetadata(GribMetadata):
+    """Represent standalone GRIB metadata owning an ecCodes GRIB handle.
+
+    :class:`StandAloneGribMetadata` possesses its own ecCodes handle. Calling
+    :meth:`override` on :obj:`GribMetadata` always returns a
+    :class:`StandAloneGribMetadata` object.
+
+    >>> ds = earthkit.data.from_source("file", "docs/examples/test4.grib")
+    >>> md = ds[0].metadata()
+    >>> md["shortName"]
+    't'
+    >>> md.get("shortName")
+    't'
+    >>> md.get("nonExistentKey")
+    >>> md.get("nonExistentKey", 12)
+    12
+
+    Examples
+    --------
+    :ref:`/examples/grib_metadata_object.ipynb`
+
+    """
+
+    def __init__(self, handle, **kwargs):
+        if not isinstance(handle, self._handle_type()):
+            raise TypeError(f"GribMetadata: expected handle type {self._handle_type()}, got {type(handle)}")
+        self.__handle = handle
+        self._kwargs = dict(**kwargs)
+        super().__init__(**kwargs)
+
+    @property
+    def _handle(self):
+        return self.__handle
+
     def _hide_internal_keys(self):
         return RestrictedGribMetadata(self)
 
 
 # TODO: this is a temporary solution
-class RestrictedGribMetadata(GribMetadata):
-    """Hide internal keys and namespaces in GRIB metadata"""
+class RestrictedGribMetadata(StandAloneGribMetadata):
+    """Hide internal keys and namespaces in GRIB metadata.
+
+    Examples
+    --------
+    :ref:`/examples/grib_metadata_object.ipynb`
+    """
 
     EKD_NAMESPACE = "grib"
 
@@ -531,7 +590,8 @@ class RestrictedGribMetadata(GribMetadata):
     INTERNAL_NAMESPACES = ["statistics"]
 
     def __init__(self, md):
-        super().__init__(md._handle, extra=md.extra)
+        assert isinstance(md, StandAloneGribMetadata)
+        super().__init__(md._handle, shrunk=md._shrunk, extra=md.extra)
 
     @cached_method
     def _len(self):
diff --git a/src/earthkit/data/readers/netcdf/field.py b/src/earthkit/data/readers/netcdf/field.py
index 5cdad0a..10d7415 100644
--- a/src/earthkit/data/readers/netcdf/field.py
+++ b/src/earthkit/data/readers/netcdf/field.py
@@ -216,7 +216,8 @@ class XArrayField(Field):
             + ")"
         )
 
-    def _make_metadata(self):
+    @cached_property
+    def _metadata(self):
         return XArrayMetadata(self)
 
     def to_xarray(self):
@@ -272,5 +273,6 @@ class NetCDFMetadata(XArrayMetadata):
 
 
 class NetCDFField(XArrayField):
-    def _make_metadata(self):
+    @cached_property
+    def _metadata(self):
         return NetCDFMetadata(self)
diff --git a/src/earthkit/data/sources/array_list.py b/src/earthkit/data/sources/array_list.py
index 2c53365..b56bc82 100644
--- a/src/earthkit/data/sources/array_list.py
+++ b/src/earthkit/data/sources/array_list.py
@@ -38,9 +38,6 @@ class ArrayField(Field):
         super().__init__(array_backend, raw_values_backend=array_backend, metadata=metadata)
         self._array = array
 
-    def _make_metadata(self):
-        pass
-
     def _values(self, dtype=None):
         """Native array type"""
         if dtype is None:
diff --git a/src/earthkit/data/sources/forcings.py b/src/earthkit/data/sources/forcings.py
index c48bf6d..b351c4f 100644
--- a/src/earthkit/data/sources/forcings.py
+++ b/src/earthkit/data/sources/forcings.py
@@ -235,9 +235,6 @@ class ForcingField(Field):
             metadata=ForcingMetadata(d, self.maker.field.metadata().geography),
         )
 
-    def _make_metadata(self):
-        pass
-
     def _values(self, dtype=None):
         values = self.proc(self.date)
         if dtype is not None:
diff --git a/src/earthkit/data/sources/list_of_dicts.py b/src/earthkit/data/sources/list_of_dicts.py
index 6b1d50e..2e467c0 100644
--- a/src/earthkit/data/sources/list_of_dicts.py
+++ b/src/earthkit/data/sources/list_of_dicts.py
@@ -179,9 +179,6 @@ class VirtualGribField(Field):
         else:
             return v.astype(dtype)
 
-    def _make_metadata(self):
-        pass
-
 
 class GribFromDicts(GribFieldList):
     def __init__(self, list_of_dicts, *args, **kwargs):


Test patch:
diff --git a/tests/array_fieldlist/test_numpy_fs_metadata.py b/tests/array_fieldlist/test_numpy_fs_metadata.py
index 5c5957a..eb08670 100644
--- a/tests/array_fieldlist/test_numpy_fs_metadata.py
+++ b/tests/array_fieldlist/test_numpy_fs_metadata.py
@@ -23,7 +23,7 @@ from array_fl_fixtures import load_array_fl_file  # noqa: E402
 # See grib/test_grib_metadata.py
 
 
-def test_array_fl_values_metadata():
+def test_array_fl_values_metadata_basic():
     ds, _ = load_array_fl(1)
 
     # values metadata
diff --git a/tests/core/test_metadata.py b/tests/core/test_metadata.py
index 7b34264..44b471f 100644
--- a/tests/core/test_metadata.py
+++ b/tests/core/test_metadata.py
@@ -9,11 +9,14 @@
 # nor does it submit to any jurisdiction.
 #
 
+import numpy as np
 import pytest
 
 from earthkit.data import from_source
 from earthkit.data.core.metadata import RawMetadata
-from earthkit.data.readers.grib.metadata import GribMetadata
+from earthkit.data.readers.grib.metadata import GribFieldMetadata
+from earthkit.data.readers.grib.metadata import RestrictedGribMetadata
+from earthkit.data.readers.grib.metadata import StandAloneGribMetadata
 from earthkit.data.testing import earthkit_examples_file
 from earthkit.data.testing import earthkit_test_data_file
 
@@ -104,17 +107,21 @@ def test_raw_metadata_override_with_kwarg():
 
 def test_grib_metadata_create():
     f = from_source("file", earthkit_examples_file("test.grib"))
-    md = f[0].metadata()
-    assert isinstance(md, GribMetadata)
+
+    f0 = f[0]
+    md = f0.metadata()
+    assert isinstance(md, GribFieldMetadata)
+    assert md._handle is not None
+    assert md._handle == f0._handle
 
     # cannot create from dict
     with pytest.raises(TypeError):
-        GribMetadata({"shortName": "u", "typeOfLevel": "pl", "levelist": 1000})
+        StandAloneGribMetadata({"shortName": "u", "typeOfLevel": "pl", "levelist": 1000})
 
     # cannot create from raw metadata
     raw_md = RawMetadata({"shortName": "u", "typeOfLevel": "pl", "levelist": 1000})
     with pytest.raises(TypeError):
-        GribMetadata(raw_md)
+        StandAloneGribMetadata(raw_md)
 
 
 def test_grib_metadata_get():
@@ -265,7 +272,7 @@ def test_grib_metadata_override_extra():
     assert md["shortName"] == "2t"
 
     extra = {"my_custom_key": "2", "shortName": "N", "perturbationNumber": 2}
-    md = GribMetadata(md._handle, extra=extra)
+    md = StandAloneGribMetadata(md._handle, extra=extra)
 
     assert md["my_custom_key"] == "2"
     assert md["perturbationNumber"] == 2
@@ -290,6 +297,50 @@ def test_grib_metadata_override_extra():
         break
 
 
+def test_grib_metadata_override_headers_only_true():
+    ds = from_source("file", earthkit_examples_file("test.grib"))
+    ref_size = ds[0].metadata("totalLength")
+
+    md1 = ds[0].metadata().override(headers_only_clone=True)
+    assert isinstance(md1, StandAloneGribMetadata)
+    assert md1._handle is not None
+    assert md1._handle != ds[0]._handle
+    assert md1["totalLength"] - ref_size < -10
+    assert md1._shrunk
+
+    md2 = md1._hide_internal_keys()
+    assert isinstance(md2, RestrictedGribMetadata)
+    assert md2._handle is not None
+    assert md2._handle != ds[0]._handle
+    assert md2._handle == md1._handle
+    assert md2._shrunk
+
+    with pytest.raises(KeyError):
+        md2["average"]
+
+
+def test_grib_metadata_override_headers_only_false():
+    ds = from_source("file", earthkit_examples_file("test.grib"))
+    ref_size = ds[0].metadata("totalLength")
+
+    md1 = ds[0].metadata().override(headers_only_clone=False)
+    assert isinstance(md1, StandAloneGribMetadata)
+    assert md1._handle is not None
+    assert md1._handle != ds[0]._handle
+    assert np.isclose(md1["totalLength"], ref_size)
+    assert not md1._shrunk
+
+    md2 = md1._hide_internal_keys()
+    assert isinstance(md2, RestrictedGribMetadata)
+    assert md2._handle is not None
+    assert md2._handle != ds[0]._handle
+    assert md2._handle == md1._handle
+    assert not md2._shrunk
+
+    with pytest.raises(KeyError):
+        md2["average"]
+
+
 if __name__ == "__main__":
     from earthkit.data.testing import main