Repository: ecmwf/earthkit-data
Pull request #209 (2023-10-02 14:18:33)

Merge main to develop


Patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 48052f9..7874606 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -15,7 +15,7 @@ repos:
   hooks:
   - id: isort
 - repo: https://github.com/psf/black
-  rev: 23.3.0
+  rev: 23.9.1
   hooks:
   - id: black
 - repo: https://github.com/keewis/blackdoc
diff --git a/README.md b/README.md
index 4f8cfbc..c74d27a 100644
--- a/README.md
+++ b/README.md
@@ -5,11 +5,9 @@
 [![PyPI version fury.io](https://badge.fury.io/py/earthkit-data.svg)](https://pypi.python.org/pypi/earthkit-data/)
 [![PyPI pyversions](https://img.shields.io/pypi/pyversions/earthkit-data.svg)](https://pypi.python.org/pypi/earthkit-data/)
 
-> :warning: **DISCLAIMER**
->
-> This project is **BETA** and will be **Experimental** for the foreseeable future.
-> Interfaces and functionality are likely to change, and the project itself may be scrapped.
-> **DO NOT** use this software in any project/software that is operational.
+**DISCLAIMER**
+
+> This project is in the **BETA** stage of development. Please be aware that interfaces and functionality may change as the project develops. If this software is to be used in operational systems you are **strongly advised to use a released tag in your system configuration**, and you should be willing to accept incoming changes and bug fixes that require adaptations on your part. ECMWF **does use** this software in operations and abides by the same caveats.
 
 A format-agnostic interface for geospatial data with a focus on meteorology and
 climate science.
diff --git a/docs/examples.rst b/docs/examples.rst
index d061b41..577fe3a 100644
--- a/docs/examples.rst
+++ b/docs/examples.rst
@@ -15,6 +15,8 @@ Here is a list of example notebooks to illustrate how to use earthkit-data.
     examples/cds.ipynb
     examples/ecmwf_open_data.ipynb
     examples/fdb.ipynb
+    examples/mars.ipynb
+    examples/polytope.ipynb
     examples/wekeo.ipynb
 
 
diff --git a/docs/examples/polytope.ipynb b/docs/examples/polytope.ipynb
new file mode 100644
index 0000000..0108e52
--- /dev/null
+++ b/docs/examples/polytope.ipynb
@@ -0,0 +1,188 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "efdd065e-e9fc-494d-9f73-9cb3d525dc4a",
+   "metadata": {},
+   "source": [
+    "## Retrieving data with polytope"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "75079b34-78ba-4536-8498-4dc3d0c3e646",
+   "metadata": {},
+   "source": [
+    "The “polytope” data source provides access to the [Polytope web services](https://polytope-client.readthedocs.io/en/latest/)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "621c0aa7-db93-441a-ab55-743fa6fbcd51",
+   "metadata": {},
+   "source": [
+    "The following example retrieves data from the ECMWF MARS archive using polytope. The dataset was prepared for the OGC GeoDataCubes working group, see details [here](https://github.com/ecmwf/ogc-gdc-usecase/tree/main)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "7910ac60-a503-4392-a719-0b780625346f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "2023-09-29 18:03:52 - INFO - Sending request...\n",
+      "{'request': 'class: rd\\n'\n",
+      "            \"date: '20200915'\\n\"\n",
+      "            'domain: g\\n'\n",
+      "            'expver: hsvs\\n'\n",
+      "            \"levellist: '1'\\n\"\n",
+      "            'levtype: pl\\n'\n",
+      "            \"param: '129.128'\\n\"\n",
+      "            'step: 0/12\\n'\n",
+      "            'stream: oper\\n'\n",
+      "            'time: 00:00:00\\n'\n",
+      "            'type: fc\\n',\n",
+      " 'verb': 'retrieve'}\n",
+      "2023-09-29 18:03:53 - INFO - Request accepted. Please poll http://polytope.ecmwf.int/api/v1/requests/5af79420-5e06-477d-8167-a54e0de84fe1 for status\n",
+      "2023-09-29 18:03:53 - INFO - Checking request status (5af79420-5e06-477d-8167-a54e0de84fe1)...\n",
+      "2023-09-29 18:03:54 - INFO - The current status of the request is 'processing'\n",
+      "2023-09-29 18:03:58 - INFO - The current status of the request is 'processed'\n",
+      "                                                                                                                                                                   \r"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>centre</th>\n",
+       "      <th>shortName</th>\n",
+       "      <th>typeOfLevel</th>\n",
+       "      <th>level</th>\n",
+       "      <th>dataDate</th>\n",
+       "      <th>dataTime</th>\n",
+       "      <th>stepRange</th>\n",
+       "      <th>dataType</th>\n",
+       "      <th>number</th>\n",
+       "      <th>gridType</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>ecmf</td>\n",
+       "      <td>z</td>\n",
+       "      <td>isobaricInhPa</td>\n",
+       "      <td>1</td>\n",
+       "      <td>20200915</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>fc</td>\n",
+       "      <td>0</td>\n",
+       "      <td>sh</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>ecmf</td>\n",
+       "      <td>z</td>\n",
+       "      <td>isobaricInhPa</td>\n",
+       "      <td>1</td>\n",
+       "      <td>20200915</td>\n",
+       "      <td>0</td>\n",
+       "      <td>12</td>\n",
+       "      <td>fc</td>\n",
+       "      <td>0</td>\n",
+       "      <td>sh</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "  centre shortName    typeOfLevel  level  dataDate  dataTime stepRange  \\\n",
+       "0   ecmf         z  isobaricInhPa      1  20200915         0         0   \n",
+       "1   ecmf         z  isobaricInhPa      1  20200915         0        12   \n",
+       "\n",
+       "  dataType  number gridType  \n",
+       "0       fc       0       sh  \n",
+       "1       fc       0       sh  "
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import earthkit.data\n",
+    "\n",
+    "request = {\n",
+    "    'stream': 'oper',\n",
+    "    'levtype': 'pl',\n",
+    "    'levellist': '1',\n",
+    "    'param': '129.128',\n",
+    "    'step': '0/12',\n",
+    "    'time': '00:00:00',\n",
+    "    'date': '20200915',\n",
+    "    'type': 'fc',\n",
+    "    'class': 'rd',\n",
+    "    'expver': 'hsvs',\n",
+    "    'domain': 'g'\n",
+    "}\n",
+    "\n",
+    "\n",
+    "ds = earthkit.data.from_source(\"polytope\", \"ecmwf-mars\", request)\n",
+    "ds.ls()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "32cb0e10-7545-4758-b2f0-99984f01d71f",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "dev",
+   "language": "python",
+   "name": "dev"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/guide/caching.rst b/docs/guide/caching.rst
index f9c315f..c6decde 100644
--- a/docs/guide/caching.rst
+++ b/docs/guide/caching.rst
@@ -11,8 +11,8 @@ Caching
 Purpose
 -------
 
-eartkit-data caches most of the remote data access on a local cache. Running again
-``earthkit.data.from_source`` will use the cached data instead of
+earthkit-data caches most of the remote data access on a local cache. Running again
+:func:`from_source` will use the cached data instead of
 downloading it again. When the cache is full, cached data is deleted according it cache policy
 (i.e. oldest data is deleted first).
 earthkit-data cache configuration is managed through the :doc:`settings`.
@@ -26,49 +26,94 @@ earthkit-data cache configuration is managed through the :doc:`settings`.
     through using mirrors.
 
 .. _cache_location:
+.. _cache_policies:
 
-Cache location
---------------
+Cache policies and locations
+------------------------------
 
-  The cache location is defined by the ``cache‑directory`` setting. Its default
-  value depends on your system:
+The primary key to control the cache in the settings is ``cache‑policy``, which can take the following values:
 
-    - ``/tmp/earthkit-data-$USER`` for Linux,
-    - ``C:\\Users\\$USER\\AppData\\Local\\Temp\\earthkit-data-$USER`` for Windows
-    - ``/tmp/.../earthkit-data-$USER`` for MacOS
+  - "user" (default)
+  - "temporary"
+  - "off"
 
+The cache location can be read and modified with Python (see the details below).
 
-  The cache location can be read and modified either with shell command or within python.
+.. tip::
 
-  .. note::
+   See the :ref:`/examples/cache.ipynb` notebook for examples.
 
-    It is recommended to restart your Jupyter kernels after changing
-    the cache location.
+.. note::
+
+  It is recommended to restart your Jupyter kernels after changing
+  the cache location.
+
+User cache policy
++++++++++++++++++++
+
+When the ``cache‑policy`` is "user" the cache is created in the directory defined by the ``user-cache-directory`` settings. The user cache directory is not cleaned up on exit. So next time you start earthkit-data it will (probably) be there again. Also, when you run multiple sessions of earthkit-data under the same user they will share the same cache.
+
+The default value of the cache directory depends on your system:
+
+  - ``/tmp/earthkit-data-$USER`` for Linux,
+  - ``C:\\Users\\$USER\\AppData\\Local\\Temp\\earthkit-data-$USER`` for Windows
+  - ``/tmp/.../earthkit-data-$USER`` for MacOS
+
+
+The following code shows how to change the ``user-cache-directory`` settings:
+
+.. code:: python
+
+  >>> from earthkit.data import settings
+  >>> settings.get("user-cache-directory")  # Find the current cache directory
+  /tmp/earthkit-data-$USER
+  >>> # Change the value of the setting
+  >>> settings.set("cache-directory", "/big-disk/earthkit-data-cache")
 
+  # Python kernel restarted
 
-  From Python:
+  >>> from earthkit.data import settings
+  >>> settings.get("user-cache-directory")  # Cache directory has been modified
+  /big-disk/earthkit-data-cache
 
-  .. code:: python
+More generally, the earthkit-data settings can be read, modified, reset
+to their default values from Python,
+see the :doc:`Settings documentation <settings>`.
 
-    >>> import earthkit.data
-    >>> earthkit.data.settings.get(
-    ...     "cache-directory"
-    ... )  # Find the current cache directory
-    /tmp/earthkit-data-$USER
-    >>> # Change the value of the setting
-    >>> earthkit.data.settings.set("cache-directory", "/big-disk/earthkit-data-cache")
 
-    # Python kernel restarted
+Temporary cache policy
+++++++++++++++++++++++++
 
-    >>> import earthkit.data
-    >>> earthkit.data.settings.get(
-    ...     "cache-directory"
-    ... )  # Cache directory has been modified
-    /big-disk/earthkit-data-cache
+When the ``cache‑policy`` is "temporary" the cache will be located in a temporary directory created by ``tempfile.TemporaryDirectory``. This directory will be unique for each earthkit-data session. When the directory object goes out of scope (at the latest on exit) the cache is cleaned up. Due to the temporary nature of this directory path it cannot be queried via the :doc:`settings`, but we need to use :meth:`cache_directory` on the ``cache`` object.
+
+.. code-block:: python
+
+  >>> from earthkit.data import cache, settings
+  >>> settings.set("cache-policy", "temporary")
+  >>> cache.cache_directory()
+  '/var/folders/ng/g0zkhc2s42xbslpsywwp_26m0000gn/T/tmp_5bf5kq8'
+
+We can specify the parent directory for the the temporary cache by using the ``temporary-cache-directory-root`` settings. By default it is set to None (no parent directory specified).
+
+.. code-block:: python
+
+  >>> from earthkit.data import cache, setting
+  >>> s = {
+  ...     "cache-policy": "temporary",
+  ...     "temporary-cache-directory-root": "~/my_demo_cache",
+  ... }
+  >>> settings.set(s)
+  >>> cache.cache_directory()
+  '~/my_demo_cache/tmp0iiuvsz5'
+
+Off cache policy
+++++++++++++++++++++++++
+
+It is also possible to turn caching off completely by setting the ``cache-policy`` to “off”.
+
+.. warning::
 
-  More generally, the earthkit-data settings can be read, modified, reset
-  to their default values from python,
-  see the :doc:`Settings documentation <settings>`.
+  At the moment, when the cache is disabled none of the sources downloading data  (e.g. :ref:`data-sources-mars`) will work. On top of that the  :ref:`data-sources-file` source will not be able to handle archive input (e.g. tar, zip).
 
 Cache limits
 ------------
@@ -100,9 +145,9 @@ Maximum-cache-disk-usage
     and ``maximum-cache-disk-usage`` to ``None``.
 
 
-Caching settings default values
+Caching settings parameters
 -------------------------------
 
-.. module-output:: generate_settings_rst .*-cache-.* cache-.*
+.. module-output:: generate_settings_rst .*-cache-.* cache-.* .*-cache
 
 Other earthkit-data settings can be found :ref:`here <settings_table>`.
diff --git a/docs/guide/include/settings-2-set.py b/docs/guide/include/settings-2-set.py
deleted file mode 100644
index e87945e..0000000
--- a/docs/guide/include/settings-2-set.py
+++ /dev/null
@@ -1,8 +0,0 @@
-import earthkit.data
-
-# Change the location of the cache:
-earthkit.data.settings.set("user-cache-directory", "/big-disk/earthkit-data-cache")
-
-# Set some default plotting options (e.g. all maps will
-# be 400 pixels wide by default):
-earthkit.data.settings.set("plotting-options", width=400)
diff --git a/docs/guide/include/settings-1-get.py b/docs/guide/include/settings-get.py
similarity index 100%
rename from docs/guide/include/settings-1-get.py
rename to docs/guide/include/settings-get.py
diff --git a/docs/guide/include/settings-3-reset.py b/docs/guide/include/settings-reset.py
similarity index 100%
rename from docs/guide/include/settings-3-reset.py
rename to docs/guide/include/settings-reset.py
diff --git a/docs/guide/include/settings-set.py b/docs/guide/include/settings-set.py
new file mode 100644
index 0000000..880d96c
--- /dev/null
+++ b/docs/guide/include/settings-set.py
@@ -0,0 +1,18 @@
+import earthkit.data
+
+# Change the location of the user defined cache:
+earthkit.data.settings.set("user-cache-directory", "/big-disk/earthkit-data-cache")
+
+# Change number of download threads
+earthkit.data.settings.set("number-of-download-threads", 7)
+
+# Multiple values can be set together. The argument list
+# can be a dictionary:
+earthkit.data.settings.set(
+    {"number-of-download-threads": 7, "url-download-timeout": "1m"}
+)
+
+# Alternatively, we can use keyword arguments. However, because
+# the “-” character is not allowed in variable names in Python we have
+# to replace “-” with “_” in all the keyword arguments:
+earthkit.data.settings.set(number_of_download_threads=8, url_download_timeout="2m")
diff --git a/docs/guide/include/settings-temporary.py b/docs/guide/include/settings-temporary.py
new file mode 100644
index 0000000..b738691
--- /dev/null
+++ b/docs/guide/include/settings-temporary.py
@@ -0,0 +1,11 @@
+import earthkit.data
+
+print(earthkit.data.settings.get("number-of-download-threads"))
+
+with earthkit.data.settings.temporary():
+    earthkit.data.settings.set("number-of-download-threads", 12)
+    print(earthkit.data.settings.get("number-of-download-threads"))
+
+# Temporary settings can also be created with arguments:
+with earthkit.data.settings.temporary("number-of-download-threads", 11):
+    print(earthkit.data.settings.get("number-of-download-threads"))
diff --git a/docs/guide/settings.rst b/docs/guide/settings.rst
index 936506c..95c7fe1 100644
--- a/docs/guide/settings.rst
+++ b/docs/guide/settings.rst
@@ -6,19 +6,24 @@ Settings
 *earthkit-data* is maintaining a set of global settings which control
 its behaviour.
 
-The settings are saved in ``~/.earthkit-data/settings.yaml``. They can
-be accessed from Python.
+The settings are automatically saved in ``~/.earthkit-data/settings.yaml`` and they can
+be accessed and modified from Python.
+
+.. tip::
+
+    See the :ref:`/examples/settings.ipynb` notebook for examples.
+
 
 Accessing settings
 ------------------
 
 earthkit-data settings can be accessed using the python API:
 
-.. literalinclude:: include/settings-1-get.py
+.. literalinclude:: include/settings-get.py
 
 
 Changing settings
------------------
+------------------
 
 .. note::
 
@@ -27,7 +32,21 @@ Changing settings
 
 earthkit-data settings can be modified using the python API:
 
-.. literalinclude:: include/settings-2-set.py
+.. literalinclude:: include/settings-set.py
+
+
+Temporary settings
+------------------
+
+We can create a temporary settings (as a context manager) as a copy of the original settings. We will still refer to it as “settings”, but it is completely independent from the original object and changes are not saved into the yaml file (even when ``settings.auto_save_settings`` is True).
+
+.. literalinclude:: include/settings-temporary.py
+
+Output::
+
+    8
+    12
+    11
 
 
 Resetting settings
@@ -40,12 +59,14 @@ Resetting settings
 
 earthkit-data settings can be reset using the python API:
 
-.. literalinclude:: include/settings-3-reset.py
+.. literalinclude:: include/settings-reset.py
 
 
 .. _settings_table:
 
-Default values
---------------
+Settings parameters
+-------------------
+
+This is the list of all the settings parameters:
 
 .. module-output:: generate_settings_rst
diff --git a/docs/guide/sources.rst b/docs/guide/sources.rst
index 82fae65..2b3cc4b 100644
--- a/docs/guide/sources.rst
+++ b/docs/guide/sources.rst
@@ -36,22 +36,23 @@ We can get data from a given source by using :func:`from_source`:
       - read data from a stream
     * - :ref:`data-sources-memory`
       - read data from a memory buffer
-    * - :ref:`data-sources-mars`
-      - retrieve data from the ECMWF `MARS archive <https://confluence.ecmwf.int/display/UDOC/MARS+user+documentation>`_
-    * - :ref:`data-sources-cds`
-      - retrieve data from the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu/>`_ (CDS)
     * - :ref:`data-sources-ads`
       - retrieve data from the `Copernicus Atmosphere Data Store <https://ads.atmosphere.copernicus.eu/>`_ (ADS)
+    * - :ref:`data-sources-cds`
+      - retrieve data from the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu/>`_ (CDS)
     * - :ref:`data-sources-eod`
       - retrieve `ECMWF open data <https://www.ecmwf.int/en/forecasts/datasets/open-data>`_
     * - :ref:`data-sources-fdb`
       - retrieve data from the `Fields DataBase <https://fields-database.readthedocs.io/en/latest/>`_ (FDB)
+    * - :ref:`data-sources-mars`
+      - retrieve data from the ECMWF `MARS archive <https://confluence.ecmwf.int/display/UDOC/MARS+user+documentation>`_
+    * - :ref:`data-sources-polytope`
+      - retrieve data from the `Polytope services <https://polytope-client.readthedocs.io/en/latest/>`_
     * - :ref:`data-sources-wekeo`
       - retrieve data from `WEkEO`_ using the WEkEO grammar
     * - :ref:`data-sources-wekeocds`
       - retrieve `CDS <https://cds.climate.copernicus.eu/>`_ data stored on `WEkEO`_ using the `cdsapi`_ grammar
 
-
 ----------------------------------
 
 .. _data-sources-file:
@@ -289,47 +290,43 @@ memory
           print(f.metadata("param"))
 
 
-.. _data-sources-mars:
-
-mars
---------------
 
-.. py:function:: from_source("mars", *args, **kwargs)
-  :noindex:
+.. _data-sources-ads:
 
-  The ``mars`` source will retrieve data from the ECMWF MARS (Meteorological Archival and Retrieval System) archive. In addition
-  to data retrieval, the request specified as ``*args`` and/or ``**kwargs`` also has GRIB post-processing options such as ``grid`` and ``area`` for regridding and
-  sub-area extraction, respectively.
+ads
+---
 
-  To figure out which data you need, or discover relevant data available in MARS, see the publicly accessible `MARS catalog`_ (or this `access restricted catalog <https://apps.ecmwf.int/mars-catalogue/>`_).
+.. py:function:: from_source("ads", dataset, *args, **kwargs)
+  :noindex:
 
-  The MARS access is direct when the MARS client is installed (as at ECMWF), otherwise it will use the `web API`_. In order to use the `web API`_ you will need to register and retrieve an access token. For a more extensive documentation about MARS, please refer to the `MARS user documentation`_.
+  The ``ads`` source accesses the `Copernicus Atmosphere Data Store`_ (ADS), using the cdsapi_ package. In addition to data retrieval, ``request`` also has post-processing options such as ``grid`` and ``area`` for regridding and sub-area extraction respectively.
 
-  :param tuple *args: positional arguments specifying the request as a dict
+  :param str dataset: the name of the ADS dataset
+  :param tuple *args: specifies the request as a dict
   :param dict **kwargs: other keyword arguments specifying the request
 
-  The following example retrieves analysis GRIB data for a subarea for 2 surface parameters:
+  The following example retrieves CAMS global reanalysis GRIB data for 2 parameters:
 
   .. code-block:: python
 
       import earthkit.data
 
       ds = earthkit.data.from_source(
-          "mars",
-          {
-              "param": ["2t", "msl"],
-              "levtype": "sfc",
-              "area": [50, -50, 20, 50],
-              "grid": [2, 2],
-              "date": "2023-05-10",
-          },
+          "ads",
+          "cams-global-reanalysis-eac4",
+          variable=["particulate_matter_10um", "particulate_matter_1um"],
+          area=[50, -50, 20, 50],  # N,W,S,E
+          date="2012-12-12",
+          time="12:00",
       )
 
-  Data downloaded from MARS is stored in the :ref:`cache <caching>`.
+  Data downloaded from the ADS is stored in the the :ref:`cache <caching>`.
+
+  To access data from the ADS, you will need to register and retrieve an access token. The process is described `here <https://ads.atmosphere.copernicus.eu/api-how-to>`__. For more information, see the `ADS_knowledge base`_.
 
   Further examples:
 
-      - :ref:`/examples/mars.ipynb`
+      - :ref:`/examples/ads.ipynb`
 
 
 .. _data-sources-cds:
@@ -372,44 +369,6 @@ cds
       - :ref:`/examples/cds.ipynb`
 
 
-.. _data-sources-ads:
-
-ads
----
-
-.. py:function:: from_source("ads", dataset, *args, **kwargs)
-  :noindex:
-
-  The ``ads`` source accesses the `Copernicus Atmosphere Data Store`_ (ADS), using the cdsapi_ package. In addition to data retrieval, ``request`` also has post-processing options such as ``grid`` and ``area`` for regridding and sub-area extraction respectively.
-
-  :param str dataset: the name of the ADS dataset
-  :param tuple *args: specifies the request as a dict
-  :param dict **kwargs: other keyword arguments specifying the request
-
-  The following example retrieves CAMS global reanalysis GRIB data for 2 parameters:
-
-  .. code-block:: python
-
-      import earthkit.data
-
-      ds = earthkit.data.from_source(
-          "ads",
-          "cams-global-reanalysis-eac4",
-          variable=["particulate_matter_10um", "particulate_matter_1um"],
-          area=[50, -50, 20, 50],  # N,W,S,E
-          date="2012-12-12",
-          time="12:00",
-      )
-
-  Data downloaded from the ADS is stored in the the :ref:`cache <caching>`.
-
-  To access data from the ADS, you will need to register and retrieve an access token. The process is described `here <https://ads.atmosphere.copernicus.eu/api-how-to>`__. For more information, see the `ADS_knowledge base`_.
-
-  Further examples:
-
-      - :ref:`/examples/ads.ipynb`
-
-
 .. _data-sources-eod:
 
 ecmwf-open-data
@@ -529,6 +488,96 @@ fdb
       - :ref:`/examples/fdb.ipynb`
 
 
+.. _data-sources-mars:
+
+mars
+--------------
+
+.. py:function:: from_source("mars", *args, **kwargs)
+  :noindex:
+
+  The ``mars`` source will retrieve data from the ECMWF MARS (Meteorological Archival and Retrieval System) archive. In addition
+  to data retrieval, the request specified as ``*args`` and/or ``**kwargs`` also has GRIB post-processing options such as ``grid`` and ``area`` for regridding and
+  sub-area extraction, respectively.
+
+  To figure out which data you need, or discover relevant data available in MARS, see the publicly accessible `MARS catalog`_ (or this `access restricted catalog <https://apps.ecmwf.int/mars-catalogue/>`_).
+
+  The MARS access is direct when the MARS client is installed (as at ECMWF), otherwise it will use the `web API`_. In order to use the `web API`_ you will need to register and retrieve an access token. For a more extensive documentation about MARS, please refer to the `MARS user documentation`_.
+
+  :param tuple *args: positional arguments specifying the request as a dict
+  :param dict **kwargs: other keyword arguments specifying the request
+
+  The following example retrieves analysis GRIB data for a subarea for 2 surface parameters:
+
+  .. code-block:: python
+
+      import earthkit.data
+
+      ds = earthkit.data.from_source(
+          "mars",
+          {
+              "param": ["2t", "msl"],
+              "levtype": "sfc",
+              "area": [50, -50, 20, 50],
+              "grid": [2, 2],
+              "date": "2023-05-10",
+          },
+      )
+
+  Data downloaded from MARS is stored in the :ref:`cache <caching>`.
+
+  Further examples:
+
+      - :ref:`/examples/mars.ipynb`
+
+
+.. _data-sources-polytope:
+
+polytope
+--------
+
+.. py:function:: from_source("polytope", collection, *args, **kwargs)
+  :noindex:
+
+  The ``polytope`` source accesses the `Polytope web services <https://polytope-client.readthedocs.io/en/latest/>`_ , using the polytope-client_ package.
+
+  :param str collection: the name of the polytope collection
+  :param tuple *args: specifies the request as a dict
+  :param dict **kwargs: other keyword arguments specifying the request
+
+  The following example retrieves GRIB data from the "ecmwf-mars" polytope collection:
+
+  .. code-block:: python
+
+      import earthkit.data
+
+      request = {
+          "stream": "oper",
+          "levtype": "pl",
+          "levellist": "1",
+          "param": "130.128",
+          "step": "0/12",
+          "time": "00:00:00",
+          "date": "20200915",
+          "type": "fc",
+          "class": "rd",
+          "expver": "hsvs",
+          "domain": "g",
+      }
+
+      ds = earthkit.data.from_source("polytope", "ecmwf-mars", request)
+
+  Data downloaded from the polytope service is stored in the the :ref:`cache <caching>`. However,
+  please note that, in the current version, each call to  :func:`from_source` will download the data again.
+
+  To access data from polytope, you will need to register and retrieve an access token.
+
+  Further examples:
+
+      - :ref:`/examples/polytope.ipynb`
+
+
+
 .. _data-sources-wekeo:
 
 wekeo
@@ -629,3 +678,5 @@ wekeocds
 
 .. _WEkEO: https://www.wekeo.eu/
 .. _hda: https://pypi.org/project/hda
+
+.. _polytope-client: https://pypi.org/project/polytope-client
diff --git a/docs/index.rst b/docs/index.rst
index baa69b8..2fee62f 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -3,7 +3,7 @@ Welcome to earthkit-data's documentation
 
 .. warning::
 
-    This project is **BETA** and will be **Experimental** for the foreseeable future. Interfaces and functionality are likely to change, and the project itself may be scrapped. **DO NOT** use this software in any project/software that is operational.
+   This project is in the **BETA** stage of development. Please be aware that interfaces and functionality may change as the project develops. If this software is to be used in operational systems you are **strongly advised to use a released tag in your system configuration**, and you should be willing to accept incoming changes and bug fixes that require adaptations on your part. ECMWF **does use** this software in operations and abides by the same caveats.
 
 **earthkit-data** is a format-agnostic Python interface for geospatial data with a focus on meteorology and
 climate science.
diff --git a/docs/release_notes/version_0.4_updates.rst b/docs/release_notes/version_0.4_updates.rst
index f9ae512..890f5eb 100644
--- a/docs/release_notes/version_0.4_updates.rst
+++ b/docs/release_notes/version_0.4_updates.rst
@@ -1,9 +1,6 @@
 Version 0.4 Updates
 /////////////////////////
 
-.. warning::
-    This is an upcoming release.
-
 Version 0.4.0
 ===============
 
@@ -11,11 +8,13 @@ New features
 ++++++++++++++++
 
 - added new sources :ref:`data-sources-wekeo` and :ref:`data-sources-wekeocds` to retrieve data from `WEkEO <https://www.wekeo.eu/>`_. See the :ref:`/examples/wekeo.ipynb` notebook example.
+- added new source :ref:`data-sources-polytope` to retrieve data from the `Polytope web services <https://polytope-client.readthedocs.io/en/latest/>`_. See the :ref:`/examples/polytope.ipynb` notebook example.
 - added the ``append`` option to :meth:`FieldList.save() <data.core.fieldlist.FieldList.save>`.
 - added the ``dtype`` option to the ``to_data()``, ``to_latlon()`` and ``to_points()`` methods both on a :class:`~data.core.fieldlist.Field` or :class:`~data.core.fieldlist.FieldList`.
 - allowed access to 32-bit GRIB data values without requiring a cast in Python from 64 to 32 bits. Only works with a recent ecCodes version (ecCodes >= 2.31.0 and eccodes-python >= 1.6.0 required). In order to use this feature set ``dtype=np.float32`` in the ``to_numpy()``, ``to_data()``, ``to_latlon()`` or ``to_points()`` methods on either a :class:`~data.core.fieldlist.Field` or :class:`~data.core.fieldlist.FieldList`.
+- implemented :class:`~data.core.fieldlist.FieldList` for xarray data
 - added :meth:`~data.core.readers.csv.CSVReader.to_xarray` to csv data
-- no ecCodes installation is required when only accessing non-GRIB data
+- no ecCodes installation required any longer when we do not want to access GRIB or BUFR data
 
 
 Fixes
@@ -27,3 +26,5 @@ Fixes
 - fixed issue when could not retrieve data from the :ref:`CDS <data-sources-wekeo>` beacause the ``month`` and ``day`` request parameters were pre-filled by earthkit-data. These parameters are not pre-filled any longer.
 - fixed issue when missing values were not correctly written to GRIB output
 - fixed issue when could not read non-fieldlist type NetCDF data with :func:`from_source`
+- fixed issue when could not save NetCDF data to disk
+- fixed issue when after modifying an archive file (tar/zip) still the original cached contents was read
diff --git a/earthkit/data/readers/archive.py b/earthkit/data/readers/archive.py
index 0aa1123..491df5b 100644
--- a/earthkit/data/readers/archive.py
+++ b/earthkit/data/readers/archive.py
@@ -7,7 +7,6 @@
 # nor does it submit to any jurisdiction.
 #
 
-import datetime
 import logging
 import os
 
@@ -69,8 +68,8 @@ class ArchiveReader(Reader):
 
         try:
             r = os.stat(self.path)
-            fsize = r[6]
-            mtime = datetime.datetime.fromtimestamp(r[8])
+            fsize = r.st_size
+            mtime = r.st_mtime_ns
         except Exception:
             fsize = 0
             mtime = 0
diff --git a/earthkit/data/readers/bufr/bufr.py b/earthkit/data/readers/bufr/bufr.py
index e455724..87ae8b8 100644
--- a/earthkit/data/readers/bufr/bufr.py
+++ b/earthkit/data/readers/bufr/bufr.py
@@ -631,14 +631,12 @@ class MultiBUFRList(BUFRList, MultiIndex):
 class BUFRInFiles(BUFRList):
     # Remote BUFRLists (with urls) are also here,
     # as the actual fieldlist is accessed on a file in cache.
-    # This class changes the interface (_getitem__ and __len__)
+    # This class changes the interface (__getitem__ and __len__)
     # into the interface (part and number_of_parts).
-    def __getitem__(self, n):
+    def _getitem(self, n):
         if isinstance(n, int):
             part = self.part(n if n >= 0 else len(self) + n)
             return BUFRMessage(part.path, part.offset, part.length)
-        else:
-            return super().__getitem__(n)
 
     def __len__(self):
         return self.number_of_parts()
diff --git a/earthkit/data/readers/grib/index/__init__.py b/earthkit/data/readers/grib/index/__init__.py
index ebe05fb..4b7cf6c 100644
--- a/earthkit/data/readers/grib/index/__init__.py
+++ b/earthkit/data/readers/grib/index/__init__.py
@@ -193,14 +193,12 @@ class GribMultiFieldList(GribFieldList, MultiIndex):
 class GribFieldListInFiles(GribFieldList):
     # Remote FieldLists (with urls) are also here,
     # as the actual fieldlist is accessed on a file in cache.
-    # This class changes the interface (_getitem__ and __len__)
+    # This class changes the interface (__getitem__ and __len__)
     # into the interface (part and number_of_parts).
-    def __getitem__(self, n):
+    def _getitem(self, n):
         if isinstance(n, int):
             part = self.part(n if n >= 0 else len(self) + n)
             return GribField(part.path, part.offset, part.length)
-        else:
-            return super().__getitem__(n)
 
     def __len__(self):
         return self.number_of_parts()
diff --git a/earthkit/data/readers/netcdf.py b/earthkit/data/readers/netcdf.py
index fa8bf0e..a94b64f 100644
--- a/earthkit/data/readers/netcdf.py
+++ b/earthkit/data/readers/netcdf.py
@@ -506,11 +506,9 @@ class XArrayFieldList(XArrayFieldListCore):
         self.FIELD_TYPE = XArrayField
         super().__init__(ds, **kwargs)
 
-    def __getitem__(self, n):
+    def _getitem(self, n):
         if isinstance(n, int):
             return self.fields[n]
-        else:
-            return super().__getitem__(n)
 
     def __len__(self):
         return len(self.fields)
@@ -621,11 +619,9 @@ class NetCDFFieldListInOneFile(NetCDFFieldListInFiles):
         assert isinstance(path, str), path
         super().__init__(path, **kwargs)
 
-    def __getitem__(self, n):
+    def _getitem(self, n):
         if isinstance(n, int):
             return self.fields[n]
-        else:
-            return super().__getitem__(n)
 
     def __len__(self):
         return len(self.fields)
diff --git a/earthkit/data/sources/numpy_list.py b/earthkit/data/sources/numpy_list.py
index aeb1473..3502a84 100644
--- a/earthkit/data/sources/numpy_list.py
+++ b/earthkit/data/sources/numpy_list.py
@@ -144,11 +144,9 @@ class ListMerger:
 
 
 class NumpyFieldList(NumpyFieldListCore):
-    def __getitem__(self, n):
+    def _getitem(self, n):
         if isinstance(n, int):
             return NumpyField(self._array[n], self._metadata[n])
-        else:
-            return super().__getitem__(n)
 
     def __len__(self):
         return (
diff --git a/earthkit/data/sources/polytope.py b/earthkit/data/sources/polytope.py
new file mode 100644
index 0000000..2a03fc1
--- /dev/null
+++ b/earthkit/data/sources/polytope.py
@@ -0,0 +1,89 @@
+# (C) Copyright 2020 ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+#
+
+import logging
+
+from . import Source
+from .multi_url import MultiUrl
+from .prompt import APIKeyPrompt
+
+LOG = logging.getLogger(__name__)
+
+
+class PolytopeWebKeyPrompt(APIKeyPrompt):
+    register_or_sign_in_url = ("",)
+    retrieve_api_key_url = ("",)
+    prompts = [
+        dict(
+            name="user_email",
+            title="Your email",
+        ),
+        dict(
+            name="user_key",
+            example="b295aad8af30332fad2fa8c963ab7900",
+            title="API key",
+            hidden=True,
+            validate="[0-9a-z]{32}",
+        ),
+    ]
+
+    rcfile = "~/.polytopeapirc"
+
+
+class Polytope(Source):
+    """
+    Retrieve data using the Polytope Web API.
+    See polytope-client.readthedocs.io for more information.
+
+    Parameters
+    ----------
+    dataset : str
+        The name of the dataset to query.
+    request: dict[str, str]
+        A collection of key : value pairs specifying the dataset.
+
+    Examples
+    --------
+    >>> src = earthkit.data.from_source("polytope", "ecmwf-mars", request)
+    >>> src.to_pandas()  # if tabular data
+    >>> src.to_xarray()  # if datacube
+    """
+
+    def __init__(self, dataset, request) -> None:
+        try:
+            import polytope
+        except ImportError:
+            raise ImportError(
+                "Polytope Web Client must be installed with 'pip install polytope-client'"
+            )
+
+        super().__init__()
+        assert isinstance(dataset, str)
+
+        self.request = dict(dataset=dataset, request=request)
+
+        credentials = PolytopeWebKeyPrompt().check(load=True)
+        self.client = polytope.api.Client(**credentials)
+
+    def __repr__(self) -> str:
+        return f"{self.__class__.__name__}({self.request['dataset']}, {self.request['request']})"
+
+    def mutate(self) -> Source:
+        pointers = self.client.retrieve(
+            self.request["dataset"],
+            self.request["request"],
+            pointer=True,
+            asynchronous=False,
+        )
+
+        urls = [p["location"] for p in pointers]
+        return MultiUrl(urls)
+
+
+source = Polytope
diff --git a/environment.yml b/environment.yml
index 2637183..148c41a 100644
--- a/environment.yml
+++ b/environment.yml
@@ -25,6 +25,7 @@ dependencies:
   - git+https://github.com/ecmwf/multiurl
   - git+https://github.com/ecmwf/pyfdb
   - ecmwf-opendata>=0.1.2
+  - polytope-client>=0.7.1
 - tqdm
 - markdown
 - make
@@ -36,8 +37,9 @@ dependencies:
 - pytest-cov
 - pytest-forked
 - pytest-timeout
-- sphinx
-- sphinx-autoapi
+- sphinx>=7.2.6
+- pip:
+  - sphinx-autoapi>=3.0.0
 - sphinx_rtd_theme
 - sphinxcontrib-apidoc
 - nbformat
diff --git a/setup.cfg b/setup.cfg
index 1a8ff01..15ec8d1 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -26,6 +26,7 @@ install_requires =
     eccodes>=1.5.0
     ecmwf-api-client>=1.6.1
     ecmwf-opendata>=0.1.2
+    polytope-client>=0.7.1
     dask
     entrypoints
     filelock


Test patch:
diff --git a/earthkit/data/testing.py b/earthkit/data/testing.py
index 7dd8904..013f3ff 100644
--- a/earthkit/data/testing.py
+++ b/earthkit/data/testing.py
@@ -97,6 +97,8 @@ try:
 except Exception:
     NO_FDB = True
 
+NO_POLYTOPE = not os.path.exists(os.path.expanduser("~/.polytopeapirc"))
+
 
 def MISSING(*modules):
     return not modules_installed(*modules)
diff --git a/tests/core/test_cache.py b/tests/core/test_cache.py
index 97bf364..949611a 100644
--- a/tests/core/test_cache.py
+++ b/tests/core/test_cache.py
@@ -198,7 +198,7 @@ def test_cache_with_log_debug(caplog):
     # the problem still occurs!
 
 
-def test_cache_zip_file_overwritten():
+def test_cache_zip_file_overwritten_1():
     with temp_directory() as tmp_dir:
         import shutil
         import zipfile
@@ -253,7 +253,9 @@ def test_cache_zip_file_changed_modtime():
         ds_path = ds.path
 
         # second pass - changed modtime
-        os.utime(zip_path, (10, 10))
+        # TODO: here we have to assume more than 1 ns passed since the
+        # zip file was created.
+        os.utime(zip_path, None)
         ds2 = from_source("file", zip_path)
         assert len(ds2) == 2
         assert ds2.path != ds_path
diff --git a/tests/documentation/test_examples.py b/tests/documentation/test_examples.py
index acd09ee..e5d2112 100644
--- a/tests/documentation/test_examples.py
+++ b/tests/documentation/test_examples.py
@@ -20,7 +20,9 @@ IGNORE = [
     "xml2rst.py",
     "actions.py",
     "generate-examples-maps.py",
-    "settings-2-set.py",
+    "settings-set.py",
+    "settings-reset.py",
+    "settings-temporary.py",
     "xref.py",
 ]
 
diff --git a/tests/documentation/test_notebooks.py b/tests/documentation/test_notebooks.py
index c0456d3..2b4f972 100644
--- a/tests/documentation/test_notebooks.py
+++ b/tests/documentation/test_notebooks.py
@@ -22,7 +22,14 @@ from earthkit.data.testing import MISSING, earthkit_file
 
 EXAMPLES = earthkit_file("docs", "examples")
 
-SKIP = ["fdb.ipynb", "mars.ipynb", "cds.ipynb", "ads.ipynb", "wekeo.ipynb"]
+SKIP = [
+    "fdb.ipynb",
+    "mars.ipynb",
+    "cds.ipynb",
+    "ads.ipynb",
+    "wekeo.ipynb",
+    "polytope.ipynb",
+]
 
 
 def notebooks_list():
diff --git a/tests/environment-unit-tests.yml b/tests/environment-unit-tests.yml
index 0822c06..8dde1b2 100644
--- a/tests/environment-unit-tests.yml
+++ b/tests/environment-unit-tests.yml
@@ -21,10 +21,12 @@ dependencies:
 - jupyterlab
 - ecmwf-api-client>=1.6.1
 - cdsapi
+- hda
 - pip:
   - git+https://github.com/ecmwf/multiurl
   - git+https://github.com/ecmwf/pyfdb
   - ecmwf-opendata>=0.1.2
+  - polytope-client>=0.7.1
 - tqdm
 - markdown
 - make
diff --git a/tests/sources/test_polytope.py b/tests/sources/test_polytope.py
new file mode 100644
index 0000000..fe3b27e
--- /dev/null
+++ b/tests/sources/test_polytope.py
@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+
+# (C) Copyright 2020 ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+#
+
+import sys
+
+import pytest
+
+from earthkit.data import from_source
+from earthkit.data.testing import NO_POLYTOPE
+
+
+def test_no_polytope_client(monkeypatch):
+    "Check that a useful message is given in the absence of the polytope-client library"
+    monkeypatch.setitem(sys.modules, "polytope", None)
+    with pytest.raises(ImportError) as excinfo:
+        from_source("polytope", None, None)
+        assert "pip install polytope-client" in str(excinfo.value)
+
+
+@pytest.mark.long_test
+@pytest.mark.download
+@pytest.mark.skipif(NO_POLYTOPE, reason="No access to Polytope Web API")
+def test_polytope_odb():
+    request = {
+        "database": "fdbdev",
+        "class": "rd",
+        "type": "oai",
+        "stream": "lwda",
+        "expver": "xxxx",
+        "obsgroup": "CONV",
+        "reportype": 16001,
+        "obstype": 15,
+        "date": [20150601, 20150602],
+        "time": "0/to/23/by/1",
+        "filter": "'select * where stationid=\"Ams01\"'",
+        "domain": "off",
+    }
+
+    src = from_source("polytope", "ichange", request)
+    df = src.to_pandas()
+    assert len(df) == 52
+
+
+@pytest.mark.long_test
+@pytest.mark.download
+@pytest.mark.skipif(NO_POLYTOPE, reason="No access to Polytope Web API")
+def test_polytope_grib():
+    request = {
+        "stream": "oper",
+        "levtype": "pl",
+        "levellist": "500",
+        "param": "129.128",
+        "step": "0/12",
+        "time": "00:00:00",
+        "date": "20200915",
+        "type": "fc",
+        "class": "rd",
+        "expver": "hsvs",
+        "domain": "g",
+    }
+
+    ds = from_source("polytope", "ecmwf-mars", request)
+
+    assert len(ds) == 2
+    assert ds.metadata("level") == [500, 500]