Repository: ecmwf/earthkit-data
Pull request #145 (2023-07-13 15:33:36)

Feature/caching policy
Related to #89. Refactored caching and also changed settings. 

The cache is used to store:
- downloaded files (e.g. from mars, url etc)
- GRIB/BUFR message offset index files
- GRIB metadata index db (sqlite)
- the earthkit-data cache sqlite db

Caching is now controlled via the following `settings` keys:

- `cache‑policy`: "user", "temporary", "off". Default "user".
       - "user": cache directory is defined by `user‑cache‑directory`. It is always created but not cleaned up automatically. Repeated or simultaneous earthkit-data sessions (e.g. in jupyter-lab) will/can use the very same cache directory.
       - "temporary": a temporary cache directory is created with `tempfile.TemporaryDirectory`, which is automatically cleaned up. A parent folder can be specified by `temporary‑cache‑directory‑root`. With this cache policy the cache directory is unique for each earthkit-data session.
       - "off": no cache is created. Methods relying on the cache will throw `RuntimeError`. E.g. no MARS retrieval is possible.
- `user‑cache‑directory`: Cache directory, which is used when `cache-policy` is "user". Default: `os.path.join(tempfile.gettempdir(), "earthkit-data-%s" % (getpass.getuser(),))`
- `temporary‑cache‑directory‑root`: Parent of the cache directory when `cache-policy` is "temporary". Default: None
- `use‑message‑position‑index‑cache`: Stores message offset index for GRIB/BUFR files in the cache. Default: False
- `maximum‑cache‑disk‑usage`:  Disk usage threshold after which earthkit-data expires older cached entries (% of the full disk capacity). Default: 95%
- `maximum‑cache‑size`: Maximum disk space used by the earthkit-data cache (ex: 100G or 2T). Default: None

The caching now uses lazy loading and the creation of the cache directory and the related objects are delayed until first use. So if we use:

```python
import earthkit.data
earthkit.data.settings("cache-policy", "off")
```

we can be sure no cache directory is created/accessed/monitored even if the default policy is "user" or "temporary".

For details see the notebook examples:
- https://earthkit-data.readthedocs.io/en/feature-caching-policy/examples/settings.html
- https://earthkit-data.readthedocs.io/en/feature-caching-policy/examples/cache.html


Patch:
diff --git a/docs/examples.rst b/docs/examples.rst
index 818a4c7..862e765 100644
--- a/docs/examples.rst
+++ b/docs/examples.rst
@@ -11,6 +11,7 @@ Here is a list of example notebooks to illustrate how to use earthkit-data.
     :caption: Retrieve
 
     examples/mars.ipynb
+    examples/ads.ipynb
     examples/cds.ipynb
     examples/ecmwf_open_data.ipynb
     examples/fdb.ipynb
@@ -66,4 +67,6 @@ Here is a list of example notebooks to illustrate how to use earthkit-data.
     :caption: Miscellaneous
     :glob:
 
+    examples/settings.ipynb
+    examples/cache.ipynb
     examples/projection.ipynb
diff --git a/docs/examples/ads.ipynb b/docs/examples/ads.ipynb
new file mode 100644
index 0000000..3196ed9
--- /dev/null
+++ b/docs/examples/ads.ipynb
@@ -0,0 +1,219 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7ec282fb-fdf9-45ba-b3f7-3d46ef212e98",
+   "metadata": {},
+   "source": [
+    "## Retrieving CAMS data from the ADS"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "52ed2e76-9c54-4a85-b932-6604efc7ae85",
+   "metadata": {},
+   "source": [
+    "The \"ads\" data source provides access to the [Copernicus Atmosphere Data Store (ADS)](https://ads.atmosphere.copernicus.eu)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f24509e0-dd8c-4032-9a3c-5a963ad544d9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import earthkit.data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9f4ce2c9-aaa3-4b76-aacb-1f597df3e338",
+   "metadata": {},
+   "source": [
+    "### Getting GRIB data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "be1b53fd-4871-4ebf-ae6c-a62fd1117082",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ds = earthkit.data.from_source(\n",
+    "    \"ads\",\n",
+    "    \"cams-global-reanalysis-eac4\",\n",
+    "    variable=[\"particulate_matter_10um\", \"particulate_matter_1um\"],\n",
+    "    area=[50, -50, 20, 50], #N,W,S,E \n",
+    "    date=\"2012-05-10\",\n",
+    "    time=\"12:00\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "09210614-fad6-46cf-915d-aae4a592d2d5",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>centre</th>\n",
+       "      <th>shortName</th>\n",
+       "      <th>typeOfLevel</th>\n",
+       "      <th>level</th>\n",
+       "      <th>dataDate</th>\n",
+       "      <th>dataTime</th>\n",
+       "      <th>stepRange</th>\n",
+       "      <th>dataType</th>\n",
+       "      <th>number</th>\n",
+       "      <th>gridType</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>ecmf</td>\n",
+       "      <td>pm10</td>\n",
+       "      <td>surface</td>\n",
+       "      <td>0</td>\n",
+       "      <td>20120510</td>\n",
+       "      <td>1200</td>\n",
+       "      <td>0</td>\n",
+       "      <td>an</td>\n",
+       "      <td>0</td>\n",
+       "      <td>regular_ll</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>ecmf</td>\n",
+       "      <td>pm1</td>\n",
+       "      <td>surface</td>\n",
+       "      <td>0</td>\n",
+       "      <td>20120510</td>\n",
+       "      <td>1200</td>\n",
+       "      <td>0</td>\n",
+       "      <td>an</td>\n",
+       "      <td>0</td>\n",
+       "      <td>regular_ll</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "  centre shortName typeOfLevel  level  dataDate  dataTime stepRange dataType  \\\n",
+       "0   ecmf      pm10     surface      0  20120510      1200         0       an   \n",
+       "1   ecmf       pm1     surface      0  20120510      1200         0       an   \n",
+       "\n",
+       "   number    gridType  \n",
+       "0       0  regular_ll  \n",
+       "1       0  regular_ll  "
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ds.ls()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eed0bfe1-eb81-4c26-a48b-5150f67bbb79",
+   "metadata": {},
+   "source": [
+    "### Getting NetCDF data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "47f368bd-64c7-4268-bacf-7f3f1aad1b24",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ds1 = earthkit.data.from_source(\n",
+    "    \"ads\",\n",
+    "    \"cams-global-reanalysis-eac4\",\n",
+    "    variable=[\"particulate_matter_10um\", \"particulate_matter_1um\"],\n",
+    "    area=[50, -50, 20, 50], #N,W,S,E \n",
+    "    date=\"2012-05-10\",\n",
+    "    time=\"12:00\",\n",
+    "    format=\"netcdf\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "71d9a8f0-5cc7-45ce-a748-44791d9d1220",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NetCDFField['pm10',[[time:0=1336651200000000000]]]\n",
+      "NetCDFField['pm1',[[time:0=1336651200000000000]]]\n"
+     ]
+    }
+   ],
+   "source": [
+    "for f in ds1:\n",
+    "    print(f)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5514f18a-0441-4995-8451-4f4fdd54d60f",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "dev",
+   "language": "python",
+   "name": "dev"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/examples/cache.ipynb b/docs/examples/cache.ipynb
new file mode 100644
index 0000000..f5ee942
--- /dev/null
+++ b/docs/examples/cache.ipynb
@@ -0,0 +1,297 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "65c27cc6-55b9-4d36-af83-c35cbd00caf3",
+   "metadata": {},
+   "source": [
+    "## Cache policies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "64c27d94-7d5b-4a0f-9e72-708be1609527",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import earthkit.data\n",
+    "from earthkit.data import settings, cache"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "17b90c39-c493-4d9a-95ec-fa13740a04ad",
+   "metadata": {},
+   "source": [
+    "earthkit-data caches most of the remote data access and some GRIB/BUFR indexing information in a local cache. Running again *from_source()* will use the cached data instead of downloading it again."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3d93c85a-3c9b-4496-990d-8612fcb82fcf",
+   "metadata": {},
+   "source": [
+    "In the examples below we will change the settings multiple times. First we ensure all the changes are temporary and no settings are saved into the configuration file. We also reset the settings to the defaults: "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "71214b97-2b64-442d-bd23-98f831b064d0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "settings.auto_save_settings = False\n",
+    "settings.reset()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "df5b047f-a34e-4a70-82d7-b7bb1fcc2fc2",
+   "metadata": {},
+   "source": [
+    "#### User defined cache directory"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "adc31ffe-78a7-4f67-b82a-aa3df81a7e1e",
+   "metadata": {},
+   "source": [
+    "The primary key to control the cache in the settings is **\"cache-policy\"**. The default value is **\"user\"**, which means that the cache is created in the directory defined by the **\"user-cache-directory\"** settings.\n",
+    "\n",
+    "The user cache directory is not cleaned up on exit. So next time you start earthkit.data it will be (probably) there again. Also, when you run multiple sessions of earthkit-data under the same user they will share the same cache. "
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "da6e1e1c-c248-4301-ab89-fea50b73bf57",
+   "metadata": {},
+   "source": [
+    "The settings tells us all the details about the cache policy and location:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "4c2afb82-c630-451f-88e5-f80c8aa404ac",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "user\n",
+      "/var/folders/ng/g0zkhc2s42xbslpsywwp_26m0000gn/T/earthkit-data-cgr\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(settings.get(\"cache-policy\"))\n",
+    "print(settings.get(\"user-cache-directory\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "68e82597-7ef4-40d4-abcb-c3d260a0425e",
+   "metadata": {},
+   "source": [
+    "The path to the current cache directory can also be queried through the **cache** object:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "44583d41-f1d3-4c28-8af5-90d659cdae31",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'/var/folders/ng/g0zkhc2s42xbslpsywwp_26m0000gn/T/earthkit-data-cgr'"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "cache.cache_directory()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f6665b3b-5176-4161-9671-b20eb3bcc483",
+   "metadata": {},
+   "source": [
+    "We are free to change the user cache directory to another path:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "ea59e447-084e-4bf4-8635-744c769a92e2",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'~/earthkit-data-demo-cache'"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "settings.set(\"user-cache-directory\", \"~/earthkit-data-demo-cache\")\n",
+    "cache.cache_directory()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f79ee6a2-3fa1-4149-a5e2-d1b0937be633",
+   "metadata": {},
+   "source": [
+    "#### Temporary cache directory"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "49e4e9a1-6f11-497e-9cd8-ac2a46b42374",
+   "metadata": {},
+   "source": [
+    "When the **\"cache-policy\"** is set to **\"temporary\"** the cache will be located in a temporary directory created by *tempfile.TemporaryDirectory*. This directory will be **unique** for each earthkit.data session. When the directory object goes out of scope (at the latest on exit) the cache is cleaned up. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "36aa1fad-2594-4a43-b115-60ac0e602a9a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'/var/folders/ng/g0zkhc2s42xbslpsywwp_26m0000gn/T/tmp_5bf5kq8'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "settings.set(\"cache-policy\", \"temporary\")\n",
+    "cache.cache_directory()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "1c0adeae-ab1f-4c6b-a4ca-ee513410fc62",
+   "metadata": {},
+   "source": [
+    "We can specify the parent directory for the the temporary cache by using the **\"temporary-cache-directory-root\"** settings. By default it is set to None (no parent directory specified)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "7b057350-7fed-4204-b825-e782f93883b6",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'~/my_demo_cache/tmp0iiuvsz5'"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "s = {\"cache-policy\": \"temporary\", \n",
+    "     \"temporary-cache-directory-root\": \"~/my_demo_cache\"}\n",
+    "settings.set(s)\n",
+    "cache.cache_directory()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "013203c6-b841-4365-9e4f-976d4270b786",
+   "metadata": {},
+   "source": [
+    "#### Turning off caching"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "506d7fec-30ca-4137-918e-afb26b436732",
+   "metadata": {},
+   "source": [
+    "It is also possible to turn caching off completely by setting the **\"cache-policy\"** to **\"off\"**."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "0052928c-b8fb-4ec5-82b2-5bc16533af06",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "True"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "settings.set(\"cache-policy\", \"off\")\n",
+    "cache.cache_directory() is None"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "de64b61b-ba84-4c6d-8801-b6fbd8d782ff",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "dev",
+   "language": "python",
+   "name": "dev"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/examples/grib_indexing.ipynb b/docs/examples/grib_indexing.ipynb
index c033ff9..8b04b7e 100644
--- a/docs/examples/grib_indexing.ipynb
+++ b/docs/examples/grib_indexing.ipynb
@@ -551,7 +551,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.16"
+   "version": "3.8.17"
   }
  },
  "nbformat": 4,
diff --git a/docs/examples/settings.ipynb b/docs/examples/settings.ipynb
new file mode 100644
index 0000000..cde3027
--- /dev/null
+++ b/docs/examples/settings.ipynb
@@ -0,0 +1,396 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "53d7b7af-fc34-450c-9c39-493f3d881c49",
+   "metadata": {},
+   "source": [
+    "## Settings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "55715969-c15d-4b7f-bf0f-459591359d76",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from earthkit.data import settings"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "01b48692-a39d-4cfd-99dd-4047501edb54",
+   "metadata": {},
+   "source": [
+    "#### Settings basics"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "9f4443e7-8e50-4583-830e-31fd4f2004f3",
+   "metadata": {},
+   "source": [
+    "The settings object is loaded from the **\"~/.earthkit-data/settings.yaml\"** file. Changes are immediately saved back into this file unless we explicitly disable it or use temporary settings (see below for details)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bb06b2f7-5552-4247-843e-2e328beaabe9",
+   "metadata": {},
+   "source": [
+    "For the rest of this notebook we disable the settings autosave so the changes will not be written into our configuration file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "c5d47016-6f5b-4cbc-ac15-1d231546ac10",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "settings.auto_save_settings = False"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "27d1d708-8090-43af-8be8-5d81eec4791b",
+   "metadata": {},
+   "source": [
+    "We can display the current settings and the default values with:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "953cdbe7-e159-47cf-bb27-613f1ec3fba3",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<style type=\"text/css\">table.ek td {\n",
+       "    vertical-align: top;\n",
+       "    text-align: left !important;\n",
+       "}\n",
+       "\n",
+       "table.ek th {\n",
+       "    vertical-align: top;\n",
+       "    text-align: left !important;\n",
+       "}\n",
+       "</style><table class='ek'><tr><th>Name</th><th>Value</th><th>Default</th></tr><tr><td>cache-policy</td><td>'user'</td><td>'user'</td></tr><tr><td>check-out-of-date-urls</td><td>True</td><td>True</td></tr><tr><td>download-out-of-date-urls</td><td>False</td><td>False</td></tr><tr><td>maximum-cache-disk-usage</td><td>'95%'</td><td>'95%'</td></tr><tr><td>maximum-cache-size</td><td>None</td><td>None</td></tr><tr><td>number-of-download-threads</td><td>5</td><td>5</td></tr><tr><td>reader-type-check-bytes</td><td>64</td><td>64</td></tr><tr><td>temporary-cache-directory-root</td><td>None</td><td>None</td></tr><tr><td>url-download-timeout</td><td>'30s'</td><td>'30s'</td></tr><tr><td>use-message-position-index-cache</td><td>False</td><td>False</td></tr><tr><td>use-standalone-mars-client-when-available</td><td>True</td><td>True</td></tr><tr><td>user-cache-directory</td><td>'/var/folders/ng/g0zkhc2s42xbslpsywwp_26m0000gn/T/earthkit-data-cgr'</td><td>'/var/folders/ng/g0zkhc2s42xbslpsywwp_26m0000gn/T/earthkit-data-cgr'</td></tr><tr><td>version</td><td>'0.2.1.dev28+g10f7c90.d20230714'</td><td>''</td></tr></table>"
+      ],
+      "text/plain": [
+       "cache-policy: (user, user)\n",
+       "check-out-of-date-urls: (True, True)\n",
+       "download-out-of-date-urls: (False, False)\n",
+       "maximum-cache-disk-usage: (95%, 95%)\n",
+       "maximum-cache-size: (None, None)\n",
+       "number-of-download-threads: (5, 5)\n",
+       "reader-type-check-bytes: (64, 64)\n",
+       "temporary-cache-directory-root: (None, None)\n",
+       "url-download-timeout: (30s, 30s)\n",
+       "use-message-position-index-cache: (False, False)\n",
+       "use-standalone-mars-client-when-available: (True, True)\n",
+       "user-cache-directory: (/var/folders/ng/g0zkhc2s42xbslpsywwp_26m0000gn/T/earthkit-data-cgr, /var/folders/ng/g0zkhc2s42xbslpsywwp_26m0000gn/T/earthkit-data-cgr)\n",
+       "version: (0.2.1.dev28+g10f7c90.d20230714, )"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "settings"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "09612eaf-6cf1-4335-b288-86ac5b04ff25",
+   "metadata": {},
+   "source": [
+    "We can use **get()** to access settings values."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "c477d8a3-eef8-4f73-86f1-64bdbabd13d5",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "5"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "settings.get(\"number-of-download-threads\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c9a56a5a-87ef-445f-b052-43d820a9bcc3",
+   "metadata": {},
+   "source": [
+    "We can use **set()** to change values."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "373e772d-52ac-4166-94a5-b7f501a4490f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "6"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "settings.set(\"number-of-download-threads\", 6)\n",
+    "settings.get(\"number-of-download-threads\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "27182b2a-f49b-406d-af81-bbd77bef3b1b",
+   "metadata": {},
+   "source": [
+    "Multiple values can be set together. The argument list can be a dictionary:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "f2c4e61b-5e91-4986-8987-21a3deeed750",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "7\n",
+      "60\n"
+     ]
+    }
+   ],
+   "source": [
+    "settings.set({\"number-of-download-threads\": 7, \"url-download-timeout\": \"1m\"})\n",
+    "print(settings.get(\"number-of-download-threads\"))\n",
+    "print(settings.get(\"url-download-timeout\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2b0e80b7-cc1d-47df-ac9c-8815581f2b87",
+   "metadata": {},
+   "source": [
+    "Alternatively, we can use keyword arguments. However, because the \"-\" character is not allowed in variable names in Python we have to replace \"-\" with \"_\" in all the keyword arguments:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "ea398b6b-5d51-42fd-ae59-76b6140f7456",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "8\n",
+      "120\n"
+     ]
+    }
+   ],
+   "source": [
+    "settings.set(number_of_download_threads=8, url_download_timeout=\"2m\")\n",
+    "print(settings.get(\"number-of-download-threads\"))\n",
+    "print(settings.get(\"url-download-timeout\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b7c8edff-9355-4144-aee9-e0a2a55f8eba",
+   "metadata": {},
+   "source": [
+    "#### Temporary settings"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "4023ce0d-4fd8-435a-92c3-7b9261895649",
+   "metadata": {},
+   "source": [
+    "We can create a temporary settings (as a context manager) as a copy of the original settings. We will still refer to it as \"settings\", but it is completely independent from the original object and changes are not saved into the yaml file (even when *settings.auto_save_settings* is True)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "5f63aca8-b609-4436-91d8-5e9b0e56a7c0",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "8\n",
+      "12\n"
+     ]
+    }
+   ],
+   "source": [
+    "with settings.temporary():\n",
+    "    print(settings.get(\"number-of-download-threads\"))\n",
+    "    settings.set(\"number-of-download-threads\", 12)\n",
+    "    print(settings.get(\"number-of-download-threads\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b1f883bd-c63b-45dd-b333-de6ed682a62f",
+   "metadata": {},
+   "source": [
+    "When we leave the context the settings are  reverted to the original one:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "f44da09d-4d5e-4ee1-babc-943e901aeb69",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "8"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "settings.get(\"number-of-download-threads\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2670d106-7e09-42ba-bc66-335d84c38905",
+   "metadata": {},
+   "source": [
+    "A temporary settings can also be created with arguments:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "15870fd7-e2a1-4674-a701-f85861e6eb8a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "11\n",
+      "8\n"
+     ]
+    }
+   ],
+   "source": [
+    "with settings.temporary(\"number-of-download-threads\", 11):\n",
+    "    print(settings.get(\"number-of-download-threads\"))\n",
+    "\n",
+    "print(settings.get(\"number-of-download-threads\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "313fbf8f-a540-449e-b340-5c46014d931c",
+   "metadata": {},
+   "source": [
+    "#### Reset to defaults"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "f53e4a57-278d-46bb-b0bb-cde58d04ba48",
+   "metadata": {},
+   "source": [
+    "The **reset()** method resets the settings to the defaults. We demonstrate it on a temporary settings:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "2cc244ba-e80f-4111-b309-6ea7bf1b8119",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "12\n",
+      "5\n"
+     ]
+    }
+   ],
+   "source": [
+    "with settings.temporary():\n",
+    "    settings.set(\"number-of-download-threads\", 12)\n",
+    "    print(settings.get(\"number-of-download-threads\"))\n",
+    "    settings.reset()\n",
+    "    print(settings.get(\"number-of-download-threads\"))\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "13f10d70-de59-4086-82d5-8d04d4c0907c",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "dev",
+   "language": "python",
+   "name": "dev"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/guide/include/settings-1-get.py b/docs/guide/include/settings-1-get.py
index e551ff4..bcd5962 100644
--- a/docs/guide/include/settings-1-get.py
+++ b/docs/guide/include/settings-1-get.py
@@ -1,7 +1,7 @@
 import earthkit.data
 
 # Access one of the settings
-cache_path = earthkit.data.settings.get("cache-directory")
+cache_path = earthkit.data.settings.get("user-cache-directory")
 print(cache_path)
 
 # If this is the last line of a Notebook cell, this
diff --git a/docs/guide/include/settings-2-set.py b/docs/guide/include/settings-2-set.py
index 5ad9495..e87945e 100644
--- a/docs/guide/include/settings-2-set.py
+++ b/docs/guide/include/settings-2-set.py
@@ -1,7 +1,7 @@
 import earthkit.data
 
 # Change the location of the cache:
-earthkit.data.settings.set("cache-directory", "/big-disk/earthkit-data-cache")
+earthkit.data.settings.set("user-cache-directory", "/big-disk/earthkit-data-cache")
 
 # Set some default plotting options (e.g. all maps will
 # be 400 pixels wide by default):
diff --git a/docs/guide/include/settings-3-reset.py b/docs/guide/include/settings-3-reset.py
index 9f6fb4b..61e9a9f 100644
--- a/docs/guide/include/settings-3-reset.py
+++ b/docs/guide/include/settings-3-reset.py
@@ -1,7 +1,7 @@
 import earthkit.data
 
 # Reset a named setting to its default value
-earthkit.data.settings.reset("cache-directory")
+earthkit.data.settings.reset("user-cache-directory")
 
 # Reset all settings to their default values
 earthkit.data.settings.reset()
diff --git a/docs/guide/sources.rst b/docs/guide/sources.rst
index c5f2947..e823475 100644
--- a/docs/guide/sources.rst
+++ b/docs/guide/sources.rst
@@ -40,6 +40,8 @@ We can get data from a given source by using :func:`from_source`:
       - retrieve data from the ECMWF `MARS archive <https://confluence.ecmwf.int/display/UDOC/MARS+user+documentation>`_
     * - :ref:`data-sources-cds`
       - retrieve data from the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu/>`_ (CDS)
+    * - :ref:`data-sources-ads`
+      - retrieve data from the `Copernicus Atmosphere Data Store <https://ads.atmosphere.copernicus.eu/>`_ (ADS)
     * - :ref:`data-sources-eod`
       - retrieve `ECMWF open data <https://www.ecmwf.int/en/forecasts/datasets/open-data>`_
     * - :ref:`data-sources-fdb`
@@ -359,13 +361,51 @@ cds
 
   Data downloaded from the CDS is stored in the the :ref:`cache <caching>`.
 
-  To access data from the CDS, you will need to register and retrieve an access token. The process is described `here <https://cds.climate.copernicus.eu/api-how-to>`__. For more information, see the CDS `knowledge base`_.
+  To access data from the CDS, you will need to register and retrieve an access token. The process is described `here <https://cds.climate.copernicus.eu/api-how-to>`__. For more information, see the `CDS_knowledge base`_.
 
   Further examples:
 
       - :ref:`/examples/cds.ipynb`
 
 
+.. _data-sources-ads:
+
+ads
+---
+
+.. py:function:: from_source("ads", dataset, *args, **kwargs)
+  :noindex:
+
+  The ``ads`` source accesses the `Copernicus Atmosphere Data Store`_ (ADS), using the cdsapi_ package. In addition to data retrieval, ``request`` also has post-processing options such as ``grid`` and ``area`` for regridding and sub-area extraction respectively.
+
+  :param str dataset: the name of the ADS dataset
+  :param tuple *args: specifies the request as a dict
+  :param dict **kwargs: other keyword arguments specifying the request
+
+  The following example retrieves CAMS global reanalysis GRIB data for 2 parameters:
+
+  .. code-block:: python
+
+      import earthkit.data
+
+      ds = earthkit.data.from_source(
+          "ads",
+          "cams-global-reanalysis-eac4",
+          variable=["particulate_matter_10um", "particulate_matter_1um"],
+          area=[50, -50, 20, 50],  # N,W,S,E
+          date="2012-12-12",
+          time="12:00",
+      )
+
+  Data downloaded from the ADS is stored in the the :ref:`cache <caching>`.
+
+  To access data from the ADS, you will need to register and retrieve an access token. The process is described `here <https://ads.atmosphere.copernicus.eu/api-how-to>`__. For more information, see the `ADS_knowledge base`_.
+
+  Further examples:
+
+      - :ref:`/examples/ads.ipynb`
+
+
 .. _data-sources-eod:
 
 ecmwf-open-data
@@ -490,7 +530,9 @@ fdb
 .. _web API: https://www.ecmwf.int/en/forecasts/access-forecasts/ecmwf-web-api
 
 .. _Copernicus Climate Data Store: https://cds.climate.copernicus.eu/
+.. _Copernicus Atmosphere Data Store: https://ads.atmosphere.copernicus.eu/
 .. _cdsapi: https://pypi.org/project/cdsapi/
-.. _knowledge base: https://confluence.ecmwf.int/display/CKB/Copernicus+Knowledge+Base
+.. _CDS_knowledge base: https://confluence.ecmwf.int/pages/viewpage.action?pageId=151530614
+.. _ADS_knowledge base: https://confluence.ecmwf.int/pages/viewpage.action?pageId=151530675
 
 .. _ECMWF open data: https://www.ecmwf.int/en/forecasts/datasets/open-data
diff --git a/earthkit/data/core/caching.py b/earthkit/data/core/caching.py
index edf8e79..1549421 100644
--- a/earthkit/data/core/caching.py
+++ b/earthkit/data/core/caching.py
@@ -29,11 +29,13 @@ import shutil
 import sqlite3
 import threading
 import time
+from abc import ABCMeta, abstractmethod
 
 import pandas as pd
 from filelock import FileLock
 
 from earthkit.data.core.settings import SETTINGS
+from earthkit.data.core.temporary import temp_directory
 from earthkit.data.utils import humanize
 from earthkit.data.utils.html import css
 
@@ -101,7 +103,7 @@ def in_executor(func):
     @functools.wraps(func)
     def wrapped(*args, **kwargs):
         global CACHE
-        s = CACHE.enqueue(func, *args, **kwargs)
+        s = CACHE._manager.enqueue(func, *args, **kwargs)
         return s.result()
 
     return wrapped
@@ -111,7 +113,7 @@ def in_executor_forget(func):
     @functools.wraps(func)
     def wrapped(*args, **kwargs):
         global CACHE
-        CACHE.enqueue(func, *args, **kwargs)
+        CACHE._manager.enqueue(func, *args, **kwargs)
         return None
 
     return wrapped
@@ -145,7 +147,7 @@ class Future:
         return self._result
 
 
-class Cache(threading.Thread):
+class CacheManager(threading.Thread):
     def __init__(self):
         super().__init__(daemon=True)
         self._connection = None
@@ -163,15 +165,20 @@ class Cache(threading.Thread):
 
     @property
     def connection(self):
-        if self._connection is None:
+        if CACHE.policy.has_cache() and self._connection is None:
             self._connection = self.new_connection()
 
         return self._connection
 
     def new_connection(self):
-        cache_dir = SETTINGS.get("cache-directory")
-        if not os.path.exists(cache_dir):
-            os.makedirs(cache_dir, exist_ok=True)
+        assert CACHE.policy.has_cache()
+        assert CACHE.policy.cache_directory() is not None
+        cache_dir = CACHE.policy.cache_directory()
+        # if not os.path.exists(cache_dir):
+        #     os.makedirs(cache_dir, exist_ok=True)
+        # cache_dir = SETTINGS.get("cache-directory")
+        # if not os.path.exists(cache_dir):
+        #     os.makedirs(cache_dir, exist_ok=True)
         cache_db = os.path.join(cache_dir, CACHE_DB)
         LOG.debug("Cache database is %s", cache_db)
         connection = sqlite3.connect(cache_db)
@@ -206,16 +213,21 @@ class Cache(threading.Thread):
             self._condition.notify_all()
             return s
 
-    def _file_in_cache_directory(self, path):
-        cache_directory = SETTINGS.get("cache-directory")
-        return path.startswith(cache_directory)
+    # def _file_in_cache_directory(self, path):
+    #     # cache_directory = self.cache_directory()
+    #     # cache_directory = SETTINGS.get("cache-directory")
+    #     return path.startswith(self._cache_directory())
 
-    def _cache_directory(self):
-        cache_directory = SETTINGS.get("cache-directory")
-        return cache_directory
+    # def _cache_directory(self):
+    #     return self._policy.cache_directory()
+
+    # def _cache_directory(self):
+    #     # cache_directory = SETTINGS.get("cache-directory")
+    #     self.policy.cache_directory()
+    #     return cache_directory
 
     def _ensure_in_cache(self, path):
-        assert self._file_in_cache_directory(path), f"File not in cache {path}"
+        assert CACHE.file_in_cache_directory(path), f"File not in cache {path}"
 
     def _settings_changed(self):
         LOG.debug("Settings changed")
@@ -317,7 +329,7 @@ class Cache(threading.Thread):
                 db.commit()
 
     def _housekeeping(self, clean=False):
-        top = SETTINGS.get("cache-directory")
+        top = CACHE.policy.cache_directory()
         with self.connection as db:
             for name in os.listdir(top):
                 if name == CACHE_DB:
@@ -444,7 +456,7 @@ class Cache(threading.Thread):
         if bytes <= 0:
             return 0
 
-        LOG.warning("CliMetLab cache: trying to free %s", humanize.bytes(bytes))
+        LOG.warning("earthkit-data cache: trying to free %s", humanize.bytes(bytes))
 
         total = 0
 
@@ -531,23 +543,26 @@ class Cache(threading.Thread):
         self._delete_entry(path)
 
     def _check_cache_size(self):
-        # Check absolute limit
-        size = self._cache_size()
-        maximum = SETTINGS.get("maximum-cache-size")
-        if maximum is not None and size > maximum:
-            self._housekeeping()
-            self._decache(size - maximum)
-
-        # Check relative limit
-        size = self._cache_size()
-        usage = SETTINGS.get("maximum-cache-disk-usage")
-        cache_directory = SETTINGS.get("cache-directory")
-        df = disk_usage(cache_directory)
-        if df.percent > usage:
-            LOG.debug("Cache disk usage %s, limit %s", df.percent, usage)
-            self._housekeeping()
-            delta = (df.percent - usage) * df.total * 0.01
-            self._decache(delta)
+        if CACHE.policy.has_cache():
+            # Check absolute limit
+            size = self._cache_size()
+            # maximum = SETTINGS.get("maximum-cache-size")
+            maximum = CACHE.policy.maximum_cache_size()
+            if maximum is not None and size > maximum:
+                self._housekeeping()
+                self._decache(size - maximum)
+
+            # Check relative limit
+            size = self._cache_size()
+            # usage = SETTINGS.get("maximum-cache-disk-usage")
+            usage = CACHE.policy.maximum_cache_disk_usage()
+            # cache_directory = SETTINGS.get("cache-directory")
+            df = disk_usage(CACHE.policy.cache_directory())
+            if df.percent > usage:
+                LOG.debug("Cache disk usage %s, limit %s", df.percent, usage)
+                self._housekeeping()
+                delta = (df.percent - usage) * df.total * 0.01
+                self._decache(delta)
 
     def _repr_html_(self):
         """Return a html representation of the cache .
@@ -567,7 +582,7 @@ class Cache(threading.Thread):
                 )  # TODO: decide how to handle temporary extension
                 if n["size"] is None:
                     n["size"] = 0
-                html.append("<table class='climetlab'>")
+                html.append("<table class='ek'>")
                 html.append("<td><td colspan='2'>%s</td></tr>" % (n["path"],))
 
                 for k in [x for x in n.keys() if x not in ("path", "owner_data")]:
@@ -598,22 +613,217 @@ class Cache(threading.Thread):
         return count, size
 
 
+class CachePolicy(metaclass=ABCMeta):
+    @staticmethod
+    def from_str(name):
+        p = _cache_policies.get(name, None)
+        if p is not None:
+            return p()
+        else:
+            raise NotImplementedError(f"Unknown cache policy={name}")
+
+    def __eq__(self, other):
+        if isinstance(other, str):
+            name = other
+            return type(self) == _cache_policies[name]
+        else:
+            return type(self) == type(other)
+
+    def _make_dir(self, path):
+        if not os.path.exists(path):
+            os.makedirs(path, exist_ok=True)
+
+    @abstractmethod
+    def has_cache(self):
+        pass
+
+    @abstractmethod
+    def cache_directory(self):
+        pass
+
+    @abstractmethod
+    def use_message_position_index_cache(self):
+        pass
+
+    @abstractmethod
+    def is_cache_size_managed(self):
+        pass
+
+    @abstractmethod
+    def maximum_cache_size(self):
+        pass
+
+    @abstractmethod
+    def maximum_cache_disk_usage(self):
+        pass
+
+
+class NoCachePolicy(CachePolicy):
+    def __del__(self):
+        pass
+
+    def has_cache(self):
+        return False
+
+    def cache_directory(self):
+        return None
+
+    def use_message_position_index_cache(self):
+        return False
+
+    def is_cache_size_managed(self):
+        return False
+
+    def maximum_cache_size(self):
+        return None
+
+    def maximum_cache_disk_usage(self):
+        return None
+
+    def __repr__(self):
+        r = f"{self.__class__.__name__}"
+        return r
+
+
+class UserCachePolicy(CachePolicy):
+    def has_cache(self):
+        return True
+
+    def cache_directory(self):
+        d = SETTINGS.get("user-cache-directory")
+        self._make_dir(d)
+        return d
+
+    def use_message_position_index_cache(self):
+        return SETTINGS.get("use-message-position-index-cache")
+
+    def is_cache_size_managed(self):
+        return (
+            self.maximum_cache_size() is not None
+            or self.maximum_cache_disk_usage() is not None
+        )
+
+    def maximum_cache_size(self):
+        return SETTINGS.get("maximum-cache-size")
+
+    def maximum_cache_disk_usage(self):
+        return SETTINGS.get("maximum-cache-disk-usage")
+
+    def __repr__(self):
+        r = [{self.__class__.__name__}]
+        r.append(f"cache-directory={self.cache_directory()}")
+        r.append(f"maximum-cache-size={self.maximum_cache_size()}")
+        r.append(f"maximum-cache-disk-usage={self.maximum_cache_disk_usage()}")
+        return "\n".join(r)
+
+
+class TmpCachePolicy(UserCachePolicy):
+    def __init__(self):
+        self._root = None
+        self._dir = None
+
+    def cache_directory(self):
+        if self._dir is None:
+            self._root = SETTINGS.get("temporary-cache-directory-root")
+            self._dir = temp_directory(dir=self._root)
+        return self._dir.path
+
+    def __eq__(self, other):
+        if super().__eq__(other):
+            return self._root == SETTINGS.get("temporary-cache-directory-root")
+        else:
+            return False
+
+
+_cache_policies = {
+    "off": NoCachePolicy,
+    "temporary": TmpCachePolicy,
+    "user": UserCachePolicy,
+}
+
+
+class Cache:
+    def __init__(self):
+        self._manager = None
+        self._policy = None
+        self._manager_methods = {}
+
+    @property
+    def policy(self):
+        self._lazy_init()
+        return self._policy
+
+    def _lazy_init(self):
+        if self._policy is None:
+            self.settings_changed()
+
+    def settings_changed(self):
+        LOG.debug(
+            "Cache settings_changed. cache-policy=" + SETTINGS.get("cache-policy")
+        )
+        if self._policy != SETTINGS.get("cache-policy"):
+            self._policy = CachePolicy.from_str(SETTINGS.get("cache-policy"))
+            if self._policy.has_cache() and self._manager is None:
+                self._manager = CacheManager()
+                self._manager.start()
+
+        self._settings_changed()
+
+    def _call_manager(self, name, forget, *args, **kwargs):
+        if self.policy.has_cache() and self._manager is not None:
+            if name not in self._manager_methods:
+                manager_fn_name = "_" + name
+                if forget:
+                    self._manager_methods[name] = in_executor_forget(
+                        getattr(self._manager, manager_fn_name)
+                    )
+                else:
+                    self._manager_methods[name] = in_executor(
+                        getattr(self._manager, manager_fn_name)
+                    )
+            return self._manager_methods[name](*args, **kwargs)
+
+    def dump_cache_database(self, *args, **kwargs):
+        return self._call_manager("dump_cache_database", False, *args, **kwargs)
+
+    def summary_dump_cache_database(self, *args, **kwargs):
+        return self._call_manager("summary_dump_cache_database", False, *args, **kwargs)
+
+    def register_cache_file(self, *args, **kwargs):
+        return self._call_manager("register_cache_file", False, *args, **kwargs)
+
+    def update_entry(self, *args, **kwargs):
+        return self._call_manager("update_entry", False, *args, **kwargs)
+
+    def decache_file(self, *args, **kwargs):
+        return self._call_manager("decache_file", False, *args, **kwargs)
+
+    def check_cache_size(self, *args, **kwargs):
+        return self._call_manager("check_cache_size", True, *args, **kwargs)
+
+    def cache_size(self, *args, **kwargs):
+        return self._call_manager("cache_size", False, *args, **kwargs)
+
+    def cache_entries(self, *args, **kwargs):
+        return self._call_manager("cache_entries", False, *args, **kwargs)
+
+    def purge_cache(self, *args, **kwargs):
+        return self._call_manager("purge_cache", False, *args, **kwargs)
+
+    def housekeeping(self, *args, **kwargs):
+        return self._call_manager("housekeeping", False, *args, **kwargs)
+
+    def _settings_changed(self, *args, **kwargs):
+        return self._call_manager("settings_changed", False, *args, **kwargs)
+
+    def cache_directory(self):
+        return self.policy.cache_directory()
+
+    def file_in_cache_directory(self, path):
+        return path.startswith(self.cache_directory())
+
+
 CACHE = Cache()
-CACHE.start()
-
-dump_cache_database = in_executor(CACHE._dump_cache_database)
-summary_dump_cache_database = in_executor(CACHE._summary_dump_cache_database)
-register_cache_file = in_executor(CACHE._register_cache_file)
-update_entry = in_executor(CACHE._update_entry)
-check_cache_size = in_executor_forget(CACHE._check_cache_size)
-cache_size = in_executor(CACHE._cache_size)
-cache_entries = in_executor(CACHE._cache_entries)
-purge_cache = in_executor(CACHE._purge_cache)
-housekeeping = in_executor(CACHE._housekeeping)
-decache_file = in_executor(CACHE._decache_file)
-file_in_cache_directory = in_executor(CACHE._file_in_cache_directory)
-settings_changed = in_executor(CACHE._settings_changed)
-cache_directory = in_executor(CACHE._cache_directory)
 
 
 def cache_file(
@@ -640,6 +850,9 @@ def cache_file(
     path : str
         Full path to the cache file.
     """
+    if not CACHE.policy.has_cache() or CACHE.cache_directory() is None:
+        raise RuntimeError("Cache is disabled. Cannot create cache file.")
+
     m = hashlib.sha256()
     m.update(owner.encode("utf-8"))
 
@@ -651,11 +864,12 @@ def cache_file(
 
     if replace is not None:
         # Don't replace files that are not in the cache
-        if not file_in_cache_directory(replace):
+        if not CACHE.file_in_cache_directory(replace):
             replace = None
 
     path = os.path.join(
-        SETTINGS.get("cache-directory"),
+        CACHE.cache_directory(),
+        # SETTINGS.get("cache-directory"),
         "{}-{}{}".format(
             owner.lower(),
             m.hexdigest(),
@@ -663,7 +877,7 @@ def cache_file(
         ),
     )
 
-    record = register_cache_file(path, owner, args)
+    record = CACHE.register_cache_file(path, owner, args)
     if os.path.exists(path):
         if callable(force):
             owner_data = record["owner_data"]
@@ -672,7 +886,7 @@ def cache_file(
             force = force(args, path, owner_data)
 
         if force:
-            decache_file(path)
+            CACHE.decache_file(path)
 
     if not os.path.exists(path):
         lock = path + ".lock"
@@ -682,12 +896,9 @@ def cache_file(
                 path
             ):  # Check again, another thread/process may have created the file
                 owner_data = create(path + ".tmp", args)
-
                 os.rename(path + ".tmp", path)
-
-                update_entry(path, owner_data)
-
-                check_cache_size()
+                CACHE.update_entry(path, owner_data)
+                CACHE.check_cache_size()
 
         try:
             os.unlink(lock)
@@ -730,4 +941,4 @@ def auxiliary_cache_file(
 
 
 # housekeeping()
-SETTINGS.on_change(settings_changed)
+SETTINGS.on_change(CACHE.settings_changed)
diff --git a/earthkit/data/core/settings.py b/earthkit/data/core/settings.py
index 64c0e26..e654d92 100644
--- a/earthkit/data/core/settings.py
+++ b/earthkit/data/core/settings.py
@@ -12,6 +12,7 @@ import getpass
 import logging
 import os
 import tempfile
+from abc import ABCMeta, abstractmethod
 from contextlib import contextmanager
 from typing import Callable
 
@@ -24,20 +25,43 @@ from earthkit.data.utils.humanize import (
     as_percent,
     as_seconds,
     interval_to_human,
+    list_to_human,
 )
 from earthkit.data.utils.interval import Interval
 
 LOG = logging.getLogger(__name__)
 
 DOT_EARTHKIT_DATA = os.path.expanduser("~/.earthkit_data")
+EARTHKIT_SETTINGS_DIR = DOT_EARTHKIT_DATA
 
 
-class Validator:
+class Validator(metaclass=ABCMeta):
+    @abstractmethod
     def check(self, value):
-        raise NotImplementedError()
+        pass
 
+    @abstractmethod
     def explain(self):
-        return str()
+        pass
+
+    @staticmethod
+    def make(value):
+        v = _validators.get(type(value), None)
+        if v is not None:
+            return v(value)
+        else:
+            raise TypeError(f"Cannot create Validator for type={type(value)}")
+
+
+class ValueValidator(Validator):
+    def __init__(self, value):
+        self.value = value
+
+    def check(self, value):
+        return value == self.value
+
+    def explain(self):
+        return f"Valid when = {self.value}."
 
 
 class IntervalValidator(Validator):
@@ -51,6 +75,20 @@ class IntervalValidator(Validator):
         return f"Valid when {interval_to_human(self.interval)}."
 
 
+class ListValidator(Validator):
+    def __init__(self, values):
+        self.values = values
+
+    def check(self, value):
+        return value in self.values
+
+    def explain(self):
+        return f"Valid values: {list_to_human(self.values)}."
+
+
+_validators = {Interval: IntervalValidator, bool: ValueValidator, list: ListValidator}
+
+
 class Setting:
     def __init__(
         self,
@@ -88,42 +126,49 @@ class Setting:
     @property
     def docs_description(self):
         d = self.description
+        t = ""
         if self.validator:
             t = self.validator.explain()
-            if t:
-                return d + " " + t
-        return d
+
+        return d.replace("{validator}", t)
+
+    def validate(self, name, value):
+        if self.validator is not None and not self.validator.check(value):
+            raise ValueError(
+                f"Settings {name} cannot be set to {value}. {self.validator.explain()}"
+            )
 
 
 _ = Setting
 
 
 SETTINGS_AND_HELP = {
-    "cache-directory": _(
+    "user-cache-directory": _(
         os.path.join(tempfile.gettempdir(), "earthkit-data-%s" % (getpass.getuser(),)),
-        """Directory of where the dowloaded files are cached, with ``${USER}`` is the user id.
+        """Cache directory used when ``cache-policy`` is ``user``.
         See :doc:`/guide/caching` for more information.""",
         docs_default=os.path.join("TMP", "earthkit-data-%s" % (getpass.getuser(),)),
     ),
-    "dask-directories": _(
-        [os.path.join(DOT_EARTHKIT_DATA, "dask")],
-        """List of directories where to search for dask cluster definitions.
-        See :ref:`dask` for more information.""",
-    ),
-    "datasets-directories": _(
-        [os.path.join(DOT_EARTHKIT_DATA, "datasets")],
-        """List of directories where to search for datasets definitions.
-        See :ref:`datasets` for more information.""",
-    ),
-    "datasets-catalogs-urls": _(
-        ["https://github.com/ecmwf-lab/climetlab-datasets/raw/main/datasets"],
-        """List of url where to search for catalogues of datasets definitions.
-        See :ref:`datasets` for more information.""",
+    "temporary-cache-directory-root": _(
+        None,
+        """Parent of the cache directory when ``cache-policy`` is ``temporary``.
+        See :doc:`/guide/caching` for more information.""",
+        getter="_as_str",
+        none_ok=True,
     ),
     "number-of-download-threads": _(
         5,
         """Number of threads used to download data.""",
     ),
+    "cache-policy": _(
+        "user",
+        """Caching policy. {validator} See :doc:`/guide/caching` for more information. """,
+        validator=ListValidator(["off", "temporary", "user"]),
+    ),
+    "use-message-position-index-cache": _(
+        False,
+        "Stores message offset index for GRIB/BUFR files in the cache.",
+    ),
     "maximum-cache-size": _(
         None,
         """Maximum disk space used by the earthkit-data cache (ex: 100G or 2T).""",
@@ -156,7 +201,7 @@ SETTINGS_AND_HELP = {
     ),
     "reader-type-check-bytes": _(
         64,
-        "Number of bytes read from the beginning of a source to identify its type.",
+        "Number of bytes read from the beginning of a source to identify its type. {validator}",
         validator=IntervalValidator(Interval(8, 4096)),
     ),
 }
@@ -170,7 +215,9 @@ for k, v in SETTINGS_AND_HELP.items():
 
 @contextmanager
 def new_settings(s):
+    """Context manager to create new settings"""
     SETTINGS._stack.append(s)
+    SETTINGS._notify()
     try:
         yield None
     finally:
@@ -210,6 +257,8 @@ def save_settings(path, settings):
 
 
 class Settings:
+    _auto_save_settings = True
+
     def __init__(self, settings_yaml: str, defaults: dict, callbacks=[]):
         self._defaults = defaults
         self._settings = dict(**defaults)
@@ -222,9 +271,12 @@ class Settings:
     def get(self, name: str, default=NONE):
         """[summary]
 
-        Args:
-            name (str): [description]
-            default ([type], optional): [description]. Defaults to NONE.
+        Parameters
+        ----------
+            name: str
+                [description]
+            default: [type]
+                [description]. Defaults to NONE.
 
         Returns
         -------
@@ -233,9 +285,11 @@ class Settings:
         if name not in SETTINGS_AND_HELP:
             raise KeyError("No setting name '%s'" % (name,))
 
+        settings_item = SETTINGS_AND_HELP[name]
+
         getter, none_ok = (
-            SETTINGS_AND_HELP[name].getter,
-            SETTINGS_AND_HELP[name].none_ok,
+            settings_item.getter,
+            settings_item.none_ok,
         )
         if getter is None:
             getter = lambda name, value, none_ok: value  # noqa: E731
@@ -248,18 +302,45 @@ class Settings:
         return getter(name, self._settings.get(name, default), none_ok)
 
     @forward
-    def set(self, name: str, *args, **kwargs):
+    def set(self, *args, **kwargs):
+        """[summary]
+
+        Parameters
+        ----------
+            name: str
+                [description]
+            value: [type]
+                [description]
+        """
+        if len(args) == 0:
+            assert len(kwargs) > 0
+            for k, v in kwargs.items():
+                self._set(k.replace("_", "-"), v)
+            self._changed()
+        else:
+            if len(args) == 1 and isinstance(args[0], dict):
+                for k, v in args[0].items():
+                    self._set(k, v)
+            else:
+                self._set(*args, **kwargs)
+        self._changed()
+
+    def _set(self, name: str, *args, **kwargs):
         """[summary]
 
-        Args
-        ----
-            name (set): [description]
-            value ([type]): [description]
+        Parameters
+        ----------
+            name: str
+                [description]
+            value: [type]
+                [description]
         """
         if name not in SETTINGS_AND_HELP:
             raise KeyError("No setting name '%s'" % (name,))
 
-        klass = SETTINGS_AND_HELP[name].kind
+        settings_item = SETTINGS_AND_HELP[name]
+
+        klass = settings_item.kind
 
         if klass in (bool, int, float, str):
             # TODO: Proper exceptions
@@ -286,8 +367,8 @@ class Settings:
                 value = args[0]
 
         getter, none_ok = (
-            SETTINGS_AND_HELP[name].getter,
-            SETTINGS_AND_HELP[name].none_ok,
+            settings_item.getter,
+            settings_item.none_ok,
         )
         if getter is not None:
             assert len(args) == 1
@@ -299,25 +380,21 @@ class Settings:
             if not isinstance(value, klass):
                 raise TypeError("Setting '%s' must be of type '%s'" % (name, klass))
 
-        validator = SETTINGS_AND_HELP[name].validator
-        if validator is not None:
-            if not validator.check(value):
-                raise ValueError(
-                    f"Settings {name} cannot be set to {value}. {validator.explain()}"
-                )
-
+        settings_item.validate(name, value)
         self._settings[name] = value
-        self._changed()
+
+        LOG.debug(f"_set {name}={value} stack_size={len(self._stack)}")
 
     @forward
     def reset(self, name: str = None):
         """Reset setting(s) to default values.
 
-        Args
-        ----
-            name (str, optional): The name of the setting to reset to default. If the setting
-            does not have a default, it is removed. If `None` is passed, all settings are reset
-            to their default values. Defaults to None.
+        Parameters
+        ----------
+            name: str, optional
+                The name of the setting to reset to default. If the setting
+                does not have a default, it is removed. If `None` is passed, all settings are
+                reset to their default values. Defaults to None.
         """
         if name is None:
             self._settings = dict(**DEFAULTS)
@@ -330,14 +407,24 @@ class Settings:
                 self._settings[name] = DEFAULTS[name]
         self._changed()
 
+    @forward
+    def __repr__(self):
+        r = []
+        for k, v in sorted(self._settings.items()):
+            setting = SETTINGS_AND_HELP.get(k, None)
+            default = setting.default if setting else ""
+            r.append(f"{k}: ({v}, {default})")
+        return "\n".join(r)
+
     @forward
     def _repr_html_(self):
         html = [css("table")]
-        html.append("<table class='climetlab'>")
+        html.append("<table class='ek'>")
+        html.append("<tr><th>Name</th><th>Value</th><th>Default</th></tr>")
         for k, v in sorted(self._settings.items()):
             setting = SETTINGS_AND_HELP.get(k, None)
             default = setting.default if setting else ""
-            html.append("<tr><td>%s</td><td>%r</td><td>%r</td></td>" % (k, v, default))
+            html.append("<tr><td>%s</td><td>%r</td><td>%r</td></tr>" % (k, v, default))
         html.append("</table>")
         return "".join(html)
 
@@ -347,7 +434,8 @@ class Settings:
             yield ((k, v, SETTINGS_AND_HELP.get(k)))
 
     def _changed(self):
-        self._save()
+        if self._auto_save_settings:
+            self._save()
         self._notify()
 
     def _notify(self):
@@ -365,11 +453,20 @@ class Settings:
             save_settings(self._settings_yaml, self._settings)
         except Exception:
             LOG.error(
-                "Cannot save CliMetLab settings (%s)",
+                "Cannot save earthkit-data settings (%s)",
                 self._settings_yaml,
                 exc_info=True,
             )
 
+    def save_as(self, path):
+        try:
+            save_settings(path, self._settings)
+        except Exception:
+            LOG.error(
+                f"Cannot save earthkit-data settings ({path})",
+                exc_info=True,
+            )
+
     def _as_bytes(self, name, value, none_ok):
         return as_bytes(value, name=name, none_ok=none_ok)
 
@@ -379,16 +476,33 @@ class Settings:
     def _as_seconds(self, name, value, none_ok):
         return as_seconds(value, name=name, none_ok=none_ok)
 
+    def _as_str(self, name, value, none_ok):
+        if value is None and none_ok:
+            return None
+        return str(value)
+
     # def _as_number(self, name, value, units, none_ok):
     #     return as_number(name, value, units, none_ok)
 
     @forward
-    def temporary(self, name=None, *args, **kwargs):
-        tmp = Settings(None, self._settings, self._callbacks)
-        if name is not None:
-            tmp.set(name, *args, **kwargs)
+    def temporary(self, *args, **kwargs):
+        tmp = Settings(None, self._settings)
+        # until the tmp object is at the top of the stack we do not want
+        # notify the observers
+        if len(args) > 0 or len(kwargs) > 0:
+            # tmp does not have any callbacks so it will not broadcast the changes
+            tmp.set(*args, **kwargs)
+        tmp._callbacks = self._callbacks
         return new_settings(tmp)
 
+    @property
+    def auto_save_settings(self):
+        return Settings._auto_save_settings
+
+    @auto_save_settings.setter
+    def auto_save_settings(self, v):
+        Settings._auto_save_settings = v
+
 
 save = False
 settings_yaml = os.path.expanduser(os.path.join(DOT_EARTHKIT_DATA, "settings.yaml"))
@@ -400,7 +514,7 @@ try:
         save_settings(settings_yaml, DEFAULTS)
 except Exception:
     LOG.error(
-        "Cannot create CliMetLab settings directory, using defaults (%s)",
+        "Cannot create earthkit-data settings directory, using defaults (%s)",
         settings_yaml,
         exc_info=True,
     )
@@ -422,7 +536,7 @@ try:
 
 except Exception:
     LOG.error(
-        "Cannot load CliMetLab settings (%s), reverting to defaults",
+        "Cannot load earthkit-data settings (%s), reverting to defaults",
         settings_yaml,
         exc_info=True,
     )
diff --git a/earthkit/data/core/temporary.py b/earthkit/data/core/temporary.py
index 70d1e3a..44d0c10 100644
--- a/earthkit/data/core/temporary.py
+++ b/earthkit/data/core/temporary.py
@@ -12,7 +12,7 @@ import tempfile
 
 
 class TmpFile:
-    """The TmpFile objets are designed to be used for temporary files.
+    """The TmpFile objects are designed to be used for temporary files.
     It ensures that the file is unlinked when the object is
     out-of-scope (with __del__).
 
@@ -58,10 +58,18 @@ def temp_file(extension=".tmp") -> TmpFile:
 
 
 class TmpDirectory(tempfile.TemporaryDirectory):
+    def __init__(self, *args, **kwargs):
+        d = kwargs.get("dir", None)
+        if d == "":
+            raise ValueError("TmpDirectory: dir option cannot be an empty str!")
+        if d is not None and not os.path.exists(d):
+            os.makedirs(d)
+        super().__init__(*args, **kwargs)
+
     @property
     def path(self):
         return self.name
 
 
-def temp_directory():
-    return TmpDirectory()
+def temp_directory(dir=None):
+    return TmpDirectory(dir=dir)
diff --git a/earthkit/data/data/css/table.css b/earthkit/data/data/css/table.css
index 42388bc..2acdf91 100644
--- a/earthkit/data/data/css/table.css
+++ b/earthkit/data/data/css/table.css
@@ -1,4 +1,9 @@
-table.eh td {
+table.ek td {
+    vertical-align: top;
+    text-align: left !important;
+}
+
+table.ek th {
     vertical-align: top;
     text-align: left !important;
 }
diff --git a/earthkit/data/input_manager.py b/earthkit/data/input_manager.py
deleted file mode 100644
index dd12df5..0000000
--- a/earthkit/data/input_manager.py
+++ /dev/null
@@ -1,179 +0,0 @@
-# (C) Copyright 2021 ECMWF.
-#
-# This software is licensed under the terms of the Apache Licence Version 2.0
-# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
-# In applying this licence, ECMWF does not waive the privileges and immunities
-# granted to it by virtue of its status as an intergovernmental organisation
-# nor does it submit to any jurisdiction.
-#
-import logging
-
-from .argument import Argument
-from .transformers import AvailabilityChecker, KwargsAliasTransformer
-
-LOG = logging.getLogger(__name__)
-
-
-class InputManager:
-    def __init__(
-        self,
-        decorators,
-    ):
-        self.decorators = decorators
-        self._pipeline = None
-        self.alias_argument = None
-        self.alias_arguments = []
-        self.availabilities = []
-        self.arguments = {}
-
-    @property
-    def pipeline(self):
-        if self._pipeline is None:
-            self.build_pipeline()
-        return self._pipeline
-
-    def register_availability(self, decorator):
-        for name, values in decorator.availability.unique_values().items():
-            if name not in self.arguments:
-                self.arguments[name] = Argument(name)
-            self.arguments[name].availability = values
-        self.availabilities.append(decorator.availability)
-
-    def register_normalize(self, decorator):
-        if decorator.name not in self.arguments:
-            self.arguments[decorator.name] = Argument(decorator.name)
-        self.arguments[decorator.name].normalize = decorator.kwargs
-
-    def register_alias_argument(self, decorator):
-        self.alias_arguments.append(decorator)
-
-    def build_pipeline(self):
-        self._pipeline = []
-        LOG.debug("Building arguments from decorators:\n %s", self.decorators)
-
-        self.arguments = {}
-
-        for decorator in self.decorators:
-            decorator.register(self)
-
-        self.arguments = list(self.arguments.values())
-
-        for a in self.alias_arguments:
-            transform = KwargsAliasTransformer(a)
-            self._pipeline.append(transform)
-
-        for a in self.arguments:
-            a.add_alias_transformers(self._pipeline)
-
-        for a in self.arguments:
-            a.add_type_transformers(self._pipeline)
-
-        for a in self.arguments:
-            a.add_alias_transformers(self._pipeline)
-
-        for availability in self.availabilities:
-            transform = AvailabilityChecker(availability)
-            self._pipeline.append(transform)
-
-        for a in self.arguments:
-            a.add_format_transformers(self._pipeline)
-
-    def apply_to_kwargs_before_default(self, kwargs):
-        LOG.debug(
-            f"Apply pipeline to kwargs before resolving default values: {safe_to_str(kwargs)}"
-        )
-        for t in self.pipeline:
-            if hasattr(t, "name"):
-                LOG.debug(f" - {t.name}: apply {t}.")
-            else:
-                LOG.debug(f" - apply {t}.")
-
-            kwargs = t.execute_before_default(kwargs)
-            LOG.debug(f"       kwargs: {safe_to_str(kwargs)}")
-
-        return kwargs
-
-    def apply_to_kwargs(self, kwargs):
-        LOG.debug(f"Apply pipeline to kwargs: {safe_to_str(kwargs)}")
-        for t in self.pipeline:
-            if hasattr(t, "name"):
-                LOG.debug(f" - {t.name}: apply {t}.")
-            else:
-                LOG.debug(f" - apply {t}.")
-
-            kwargs = t.execute(kwargs)
-            LOG.debug(f"       kwargs: {safe_to_str(kwargs)}")
-
-        return kwargs
-
-    def apply_to_value(self, value):
-        for t in self.pipeline:
-            value = t.__call__(value)
-        return value
-
-    def __repr__(self) -> str:
-        txt = "ARGUMENTS:[\n"
-        txt += " availability >>>\n"
-        if self.availabilities:
-            txt += f"{self.availabilities}<<<"
-        else:
-            txt += "None<<<"
-        txt += "\n"
-        for a in self.arguments:
-            txt += f"  {a}\n"
-        txt += "]"
-        if self._pipeline is None:
-            txt += "Pipeline[not-ready]"
-        else:
-            txt += "Pipeline:[\n"
-            for t in self._pipeline:
-                txt += f"  {t}\n"
-            txt += "]"
-        return txt
-
-    def get_argument(self, name):
-        for a in self.arguments:
-            if a == name:
-                return a
-        return None
-
-    def consolidate_defaults(self, defaults):
-        for k, default in defaults.items():
-            a = self.get_argument(k)
-            if a:
-                a.set_default(default)
-
-    def apply_to_arg_kwargs(self, args, kwargs, func):
-        from earthkit.data.arguments.args_kwargs import ArgsKwargs
-
-        args_kwargs = ArgsKwargs(args, kwargs, func=func)
-        LOG.debug("Applying decorator stack (before default) to: %s %s", args, kwargs)
-        args_kwargs.kwargs = self.apply_to_kwargs_before_default(args_kwargs.kwargs)
-
-        args_kwargs.add_default_values_and_kwargs()
-        self.consolidate_defaults(args_kwargs.defaults)
-
-        LOG.debug("Applying decorator stack to: %s %s", args, kwargs)
-        args_kwargs.kwargs = self.apply_to_kwargs(args_kwargs.kwargs)
-
-        args_kwargs.ensure_positionals_only()
-
-        args, kwargs = args_kwargs.args, args_kwargs.kwargs
-
-        LOG.debug("CALLING func %s %s", args, kwargs)
-        return args, kwargs
-
-
-def normaliser(*args, **kwargs):
-    return Argument("no-name", *args, **kwargs)
-
-
-def safe_to_str(dic):
-    lst = []
-    for k, v in dic.items():
-        try:
-            v = str(v)
-        except:  # noqa: E722
-            v = "..."
-        lst.append(f"{k}={v}")
-    return ",".join(lst)
diff --git a/earthkit/data/readers/grib/index/db.py b/earthkit/data/readers/grib/index/db.py
index 52611dc..a4666df 100644
--- a/earthkit/data/readers/grib/index/db.py
+++ b/earthkit/data/readers/grib/index/db.py
@@ -12,7 +12,7 @@ import logging
 import os
 from urllib.parse import urljoin
 
-from earthkit.data.core.caching import cache_file
+from earthkit.data.core.caching import CACHE, cache_file
 from earthkit.data.readers.grib.index import FieldListInFiles
 from earthkit.data.utils import progress_bar
 
@@ -55,6 +55,9 @@ class FieldListInFilesWithDBIndex(FieldListInFiles):
             db = cls.DBCLASS(target)
             db.load_iterator(iterator)
 
+        if not CACHE.policy.has_cache():
+            raise RuntimeError("Cannot perform grib indexing when caching is disabled")
+
         db_name = cache_file(
             "grib-index",
             load,
diff --git a/earthkit/data/sources/ads.py b/earthkit/data/sources/ads.py
new file mode 100644
index 0000000..e298e44
--- /dev/null
+++ b/earthkit/data/sources/ads.py
@@ -0,0 +1,75 @@
+# (C) Copyright 2020 ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+#
+
+import os
+
+import cdsapi
+import yaml
+
+from .cds import CdsRetriever
+from .prompt import APIKeyPrompt
+
+
+class ADSAPIKeyPrompt(APIKeyPrompt):
+    register_or_sign_in_url = "https://ads.atmosphere.copernicus.eu/"
+    retrieve_api_key_url = "https://ads.atmosphere.copernicus.eu/api-how-to"
+
+    prompts = [
+        dict(
+            name="url",
+            default="https://ads.atmosphere.copernicus.eu/api/v2",
+            title="API url",
+            validate=r"http.?://.*",
+        ),
+        dict(
+            name="key",
+            example="123:abcdef01-0000-1111-2222-0123456789ab",
+            title="API key",
+            hidden=True,
+            validate=r"\d+:[\-0-9a-f]+",
+        ),
+    ]
+
+    rcfile = "~/.adsapirc"
+
+    def save(self, input, file):
+        yaml.dump(input, file, default_flow_style=False)
+
+
+def client():
+    prompt = ADSAPIKeyPrompt()
+    prompt.check()
+
+    path = os.path.expanduser(prompt.rcfile)
+
+    if not os.path.exists(path):
+        prompt.ask_user_and_save()
+
+    with open(path) as f:
+        rc = yaml.safe_load(f.read())
+
+    return cdsapi.Client(**rc)
+
+
+EXTENSIONS = {
+    "grib": ".grib",
+    "netcdf": ".nc",
+}
+
+
+class AdsRetriever(CdsRetriever):
+    sphinxdoc = """
+    AdsRetriever
+    """
+
+    def client(self):
+        return client()
+
+
+source = AdsRetriever
diff --git a/earthkit/data/sources/cds.py b/earthkit/data/sources/cds.py
index e97697c..778cd36 100644
--- a/earthkit/data/sources/cds.py
+++ b/earthkit/data/sources/cds.py
@@ -69,6 +69,9 @@ class CdsRetriever(FileSource):
     CdsRetriever
     """
 
+    def client(self):
+        return client()
+
     def __init__(self, dataset, *args, **kwargs):
         super().__init__()
 
@@ -81,7 +84,7 @@ class CdsRetriever(FileSource):
 
         requests = self.requests(**kwargs)
 
-        client()  # Trigger password prompt before thraeding
+        self.client()  # Trigger password prompt before thraeding
 
         nthreads = min(self.settings("number-of-download-threads"), len(requests))
 
@@ -96,7 +99,7 @@ class CdsRetriever(FileSource):
 
     def _retrieve(self, dataset, request):
         def retrieve(target, args):
-            client().retrieve(args[0], args[1], target)
+            self.client().retrieve(args[0], args[1], target)
 
         return self.cache_file(
             retrieve,
diff --git a/earthkit/data/sources/file.py b/earthkit/data/sources/file.py
index 15d8bfc..7c5b757 100644
--- a/earthkit/data/sources/file.py
+++ b/earthkit/data/sources/file.py
@@ -13,7 +13,7 @@ import logging
 import os
 
 from earthkit.data import from_source
-from earthkit.data.core.settings import SETTINGS
+from earthkit.data.core.caching import CACHE
 from earthkit.data.readers import reader
 from earthkit.data.sources.file_indexed import FileIndexedSource
 
@@ -126,7 +126,7 @@ class FileSource(Source, os.PathLike, metaclass=FileSourceMeta):
         return self._reader._attributes(names)
 
     def __repr__(self):
-        cache_dir = SETTINGS.get("cache-directory")
+        cache_dir = CACHE.cache_directory()
         path = getattr(self, "path", None)
         if isinstance(path, str):
             path = path.replace(cache_dir, "CACHE:")
diff --git a/earthkit/data/utils/message.py b/earthkit/data/utils/message.py
index ef3d99c..afd4edd 100644
--- a/earthkit/data/utils/message.py
+++ b/earthkit/data/utils/message.py
@@ -16,7 +16,7 @@ import time
 import eccodes
 import numpy as np
 
-from earthkit.data.core.caching import auxiliary_cache_file
+from earthkit.data.core.caching import CACHE, auxiliary_cache_file
 
 LOG = logging.getLogger(__name__)
 
@@ -59,8 +59,7 @@ class CodesMessagePositionIndex:
         self.lengths = lengths
 
     def _load(self):
-        if True:
-            # if SETTINGS.policy("message-position-cache"):
+        if CACHE.policy.use_message_position_index_cache():
             self._cache_file = auxiliary_cache_file(
                 "message-index",
                 self.path,
@@ -74,34 +73,34 @@ class CodesMessagePositionIndex:
             self._build()
 
     def _save_cache(self):
-        # assert SETTINGS.policy("message-position-cache")
-        try:
-            with open(self._cache_file, "w") as f:
-                json.dump(
-                    dict(
-                        version=self.VERSION,
-                        offsets=self.offsets,
-                        lengths=self.lengths,
-                    ),
-                    f,
-                )
-        except Exception:
-            LOG.exception("Write to cache failed %s", self._cache_file)
+        if CACHE.policy.use_message_position_index_cache():
+            try:
+                with open(self._cache_file, "w") as f:
+                    json.dump(
+                        dict(
+                            version=self.VERSION,
+                            offsets=self.offsets,
+                            lengths=self.lengths,
+                        ),
+                        f,
+                    )
+            except Exception:
+                LOG.exception("Write to cache failed %s", self._cache_file)
 
     def _load_cache(self):
-        # assert SETTINGS.policy("message-position-cache")
-        try:
-            with open(self._cache_file) as f:
-                c = json.load(f)
-                if not isinstance(c, dict):
-                    return False
-
-                assert c["version"] == self.VERSION
-                self.offsets = c["offsets"]
-                self.lengths = c["lengths"]
-                return True
-        except Exception:
-            LOG.exception("Load from cache failed %s", self._cache_file)
+        if CACHE.policy.use_message_position_index_cache():
+            try:
+                with open(self._cache_file) as f:
+                    c = json.load(f)
+                    if not isinstance(c, dict):
+                        return False
+
+                    assert c["version"] == self.VERSION
+                    self.offsets = c["offsets"]
+                    self.lengths = c["lengths"]
+                    return True
+            except Exception:
+                LOG.exception("Load from cache failed %s", self._cache_file)
 
         return False
 
diff --git a/earthkit/data/utils/module_inputs_wrapper.py b/earthkit/data/utils/module_inputs_wrapper.py
new file mode 100644
index 0000000..66e0712
--- /dev/null
+++ b/earthkit/data/utils/module_inputs_wrapper.py
@@ -0,0 +1,197 @@
+# (C) Copyright 2020 ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+#
+
+"""
+Module containing methods to transform the inputs of functions based on the function type setting,
+common signitures or mapping defined at call time
+"""
+import inspect
+import types
+import typing as T
+from ast import Module
+from functools import wraps
+
+from earthkit.data import transform
+from earthkit.data.wrappers import Wrapper
+
+try:
+    UNION_TYPES = [T.Union, types.UnionType]
+except AttributeError:
+    # types.UnionType is not in versions of python<3.9
+    UNION_TYPES = [
+        T.Union,
+    ]
+
+EMPTY_TYPES = [inspect._empty]
+
+
+def _ensure_iterable(input_item):
+    """Ensure that an item is iterable"""
+    if not isinstance(input_item, (tuple, list, dict)):
+        return [input_item]
+    return input_item
+
+
+def _ensure_tuple(input_item):
+    """Ensure that an item is a tuple"""
+    if not isinstance(input_item, tuple):
+        return tuple(_ensure_iterable(input_item))
+    return input_item
+
+
+def transform_function_inputs(
+    function: T.Callable,
+    kwarg_types: T.Dict[str, T.Any] = {},
+    convert_types: T.Union[T.Tuple[T.Any], T.Dict[str, T.Tuple[T.Any]]] = (),
+) -> T.Callable:
+    """
+    Transform the inputs to a function to match the requirements.
+    earthkit.data handles the input arg/kwarg format.
+
+    Parameters
+    ----------
+    function : Callable
+        Method to be wrapped
+    kwarg_types : Dict[str: type]
+        Mapping of accepted object types for each arg/kwarg
+    convert_types : Tuple[type]
+        List of data-types to try to convert, this can be useful when the function is versitile and can
+        accept a large number of data-types, hence only a small number of types should be converted.
+
+    Returns
+    -------
+    [type]
+        [description]
+    """
+
+    def _wrapper(kwarg_types, convert_types, *args, **kwargs):
+        kwarg_types = {**kwarg_types}
+        signature = inspect.signature(function)
+        mapping = signature_mapping(signature, kwarg_types)
+
+        # Add args to kwargs for ease of looping:
+        arg_names = []
+        for arg, name in zip(args, signature.parameters):
+            arg_names.append(name)
+            kwargs[name] = arg
+
+        convert_kwargs = [k for k in kwargs if k in mapping]
+        # Only convert some data-types, this can be used to prevent conversion for for functions which
+        #  accept a long-list of formats, e.g. numpy methods can accept xarray, pandas and more
+
+        # Filter for convert_types
+        if convert_types:
+            # Ensure convert_types is a dictionary
+            if not isinstance(convert_types, dict):
+                convert_types = {key: convert_types for key in convert_kwargs}
+
+            convert_kwargs = [
+                k
+                for k in convert_kwargs
+                if isinstance(kwargs[k], _ensure_tuple(convert_types.get(k, ())))
+            ]
+
+        # transform args/kwargs
+        for key in convert_kwargs:
+            value = kwargs[key]
+            kwarg_types = _ensure_iterable(mapping[key])
+            # Transform value if necessary
+            if type(value) not in kwarg_types:
+                for kwarg_type in kwarg_types:
+                    try:
+                        kwargs[key] = transform(value, kwarg_type)
+                    except Exception:
+                        # Transform was not possible, move to next kwarg type.
+                        # If no transform is possible, format is unchanged and we rely on function to raise
+                        # an Error.
+                        continue
+                    break
+
+        # Anything that is still a Wrapper object, expand to native data format:
+        for k, v in [(_k, _v) for _k, _v in kwargs.items() if isinstance(_v, Wrapper)]:
+            try:
+                kwargs[k] = v.data
+            except Exception:
+                pass
+
+        # Extract args from kwargs:
+        args = [kwargs.pop(name) for name in arg_names]
+        return function(*args, **kwargs)
+
+    @wraps(function)
+    def wrapper(*args, _auto_inputs_transform=True, **kwargs):
+        if not _auto_inputs_transform:  # Possibility to escape wrapping at call level
+            return function(*args, **kwargs)
+        return _wrapper(kwarg_types, convert_types, *args, **kwargs)
+
+    return wrapper
+
+
+def signature_mapping(signature, kwarg_types):
+    """
+    Map args and kwargs to object types, using hierarchical selection method:
+    1. Explicitly defined type
+    2. Based on Type setting in function
+    3. Do nothing
+    """
+    mapping = {}
+    for key, parameter in signature.parameters.items():
+        if key in kwarg_types:
+            # 1. Use explicitly defined type
+            kwarg_type = kwarg_types.get(key)
+        elif parameter.annotation not in EMPTY_TYPES:
+            # 2. Use type setting from function
+            kwarg_type = parameter.annotation
+            if T.get_origin(kwarg_type) in UNION_TYPES:
+                # Need to expand union_types to list
+                kwarg_type = T.get_args(kwarg_type)
+        else:
+            # 3. Do nothing, cannot assign None, as None is a valid type
+            continue
+        mapping[key] = kwarg_type
+    return mapping
+
+
+def transform_module_inputs(in_module, **kwargs):
+    """
+    Transform the inputs to all functions in a module.
+
+    Parameters
+    ----------
+    in_module : Module
+        Module containing funcitons which are to be wrapped with transform_function_inputs
+    kwarg_types : Dict[str: type]
+        Mapping of accepted object types for each arg/kwarg
+    convert_types : Tuple[type]
+        List of data-types to try to convert, this can be useful when the function is versitile and can
+        accept a large number of data-types, hence only a small number of types should be converted.
+    kwargs: Any
+        Any other kwargs accepted by transform_function_inputs
+
+    Returns
+    -------
+    Module
+        Version of in_module where all functions are wrapped with transform_modul_inputs
+    """
+    # wrapped_module must be different to original to prevent overriding cached module
+    wrapped_module = Module
+    for name in dir(in_module):
+        func = getattr(in_module, name)
+        # Wrap any functions that are not hidden
+        if not name.startswith("_") and isinstance(func, types.FunctionType):
+            setattr(
+                wrapped_module,
+                name,
+                transform_function_inputs(func, **kwargs),
+            )
+        else:
+            # If not a func, we just copy
+            setattr(wrapped_module, name, func)
+
+    return wrapped_module
diff --git a/earthkit/data/wrappers/ndarray.py b/earthkit/data/wrappers/ndarray.py
index 87be543..054f802 100644
--- a/earthkit/data/wrappers/ndarray.py
+++ b/earthkit/data/wrappers/ndarray.py
@@ -28,6 +28,18 @@ class NumpyNDArrayWrapper(Wrapper):
         """
         return self.data
 
+    def to_xarray(self, **kwargs):
+        """
+        Return an xarray.DataArray representation of the data.
+
+        Returns
+        -------
+        xarray.DataArray
+        """
+        import xarray as xr
+
+        return xr.DataArray(self.data, **kwargs)
+
 
 def wrapper(data, *args, **kwargs):
     import numpy as np


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 73d77a9..80036aa 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -40,3 +40,9 @@ def pytest_runtest_setup(item):
     for m in marks_in_items:
         if m in marks_to_skip:
             pytest.skip(f"test is skipped because custom pytest option: -E {flag}")
+
+    # make sure all the tests use the default settings
+    from earthkit.data import settings
+
+    settings.auto_save_settings = False
+    settings.reset()
diff --git a/tests/core/test_cache.py b/tests/core/test_cache.py
index f6f5918..deaaaf8 100644
--- a/tests/core/test_cache.py
+++ b/tests/core/test_cache.py
@@ -14,22 +14,19 @@ import os
 
 import pytest
 
-from earthkit.data import from_source, settings
-from earthkit.data.core.caching import cache_entries, cache_file, purge_cache
+from earthkit.data import cache, from_source, settings
+from earthkit.data.core.caching import cache_file
+from earthkit.data.core.temporary import temp_directory
+from earthkit.data.testing import earthkit_examples_file
 
 LOG = logging.getLogger(__name__)
 
 
-def test_cache_1():
-    with settings.temporary():
-        settings.set("maximum-cache-disk-usage", "99%")
-
-        purge_cache(matcher=lambda e: ["owner"] == "test_cache")
-
-        def touch(target, args):
-            assert args["foo"] in (1, 2)
-            with open(target, "w"):
-                pass
+def check_cache_files(dir_path):
+    def touch(target, args):
+        assert args["foo"] in (1, 2)
+        with open(target, "w"):
+            pass
 
         path1 = cache_file(
             "test_cache",
@@ -45,16 +42,27 @@ def test_cache_1():
             extension=".test",
         )
 
+        assert os.path.exists(path1)
+        assert os.path.exists(path2)
+        assert os.path.dirname(path1) == dir_path
+        assert os.path.dirname(path1) == dir_path
         assert path1 != path2
 
         cnt = 0
-        for f in cache_entries():
+        for f in cache.cache_entries():
             if f["owner"] == "test_cache":
                 cnt += 1
 
         assert cnt == 2
 
 
+def test_cache_1():
+    with settings.temporary():
+        settings.set("maximum-cache-disk-usage", "99%")
+        cache.purge_cache(matcher=lambda e: ["owner"] == "test_cache")
+        check_cache_files(settings.get("user-cache-directory"))
+
+
 # 1GB ram disk on MacOS (blocks of 512 bytes)
 # diskutil erasevolume HFS+ "RAMDisk" `hdiutil attach -nomount ram://2097152`
 @pytest.mark.skipif(not os.path.exists("/Volumes/RAMDisk"), reason="No RAM disk")
@@ -66,6 +74,109 @@ def test_cache_4():
             from_source("dummy-source", "zeros", size=100 * 1024 * 1024, n=n)
 
 
+def test_cache_policy():
+    with temp_directory() as user_dir:
+        # cache = user dir
+        with settings.temporary():
+            settings.set({"cache-policy": "user", "user-cache-directory": user_dir})
+            assert settings.get("cache-policy") == "user"
+            assert settings.get("user-cache-directory") == user_dir
+            assert cache.policy.has_cache() is True
+            cache_dir = cache.policy.cache_directory()
+            assert cache_dir == user_dir
+            assert os.path.exists(cache_dir)
+            check_cache_files(cache_dir)
+
+            # cache = temporary with auto generated path
+            with settings.temporary(
+                {"cache-policy": "temporary", "temporary-cache-directory-root": None}
+            ):
+                assert settings.get("cache-policy") == "temporary"
+                assert settings.get("temporary-cache-directory-root") is None
+                assert cache.policy.has_cache() is True
+                cache_dir = cache.policy.cache_directory()
+                assert os.path.exists(cache_dir)
+                check_cache_files(cache_dir)
+
+            # cache = user dir (again)
+            assert settings.get("cache-policy") == "user"
+            assert settings.get("user-cache-directory") == user_dir
+            assert cache.policy.has_cache() is True
+            cache_dir = cache.policy.cache_directory()
+            assert cache_dir == user_dir
+            assert os.path.exists(cache_dir)
+            check_cache_files(cache_dir)
+
+            # cache = temporary with user defined root path
+            with temp_directory() as root_dir:
+                with settings.temporary(
+                    {
+                        "cache-policy": "temporary",
+                        "temporary-cache-directory-root": root_dir,
+                    }
+                ):
+                    assert settings.get("cache-policy") == "temporary"
+                    assert settings.get("temporary-cache-directory-root") == root_dir
+                    assert cache.policy.has_cache() is True
+                    cache_dir = cache.policy.cache_directory()
+                    assert os.path.exists(cache_dir)
+                    os.path.dirname(cache_dir) == root_dir
+                    check_cache_files(cache_dir)
+
+            # cache = off
+            with settings.temporary("cache-policy", "off"):
+                assert settings.get("cache-policy") == "off"
+                assert settings.get("user-cache-directory") == user_dir
+                assert cache.policy.has_cache() is False
+                assert cache.policy.cache_directory() is None
+
+                with pytest.raises(RuntimeError):
+                    cache_file(
+                        "dummy_test_cache",
+                        None,
+                        {"foo": 1},
+                        extension=".test",
+                    )
+
+            # cache = user dir (again)
+            assert settings.get("cache-policy") == "user"
+            assert settings.get("user-cache-directory") == user_dir
+            assert cache.policy.has_cache() is True
+            cache_dir = cache.policy.cache_directory()
+            assert cache_dir == user_dir
+            assert os.path.exists(cache_dir)
+            check_cache_files(cache_dir)
+
+
+def test_url_source_no_cache():
+    with settings.temporary("cache-policy", "off"):
+        with pytest.raises(RuntimeError):
+            from_source(
+                "url",
+                "https://get.ecmwf.int/repository/test-data/earthkit-data/examples/test.grib",
+            )
+
+
+def test_grib_no_cache():
+    with settings.temporary("cache-policy", "off"):
+        ds = from_source("file", earthkit_examples_file("tuv_pl.grib"))
+        assert len(ds) == 18
+
+        f = ds[3]
+        assert f.metadata("param") == "t"
+
+
+@pytest.mark.parametrize("index_cache", [True, False])
+def test_grib_offset_index_cache(index_cache):
+    s = {"cache-policy": "temporary", "use-message-position-index-cache": index_cache}
+    with settings.temporary(s):
+        ds = from_source("file", earthkit_examples_file("tuv_pl.grib"))
+        assert len(ds) == 18
+
+        f = ds[3]
+        assert f.metadata("param") == "t", f"index-cache={index_cache}"
+
+
 if __name__ == "__main__":
     from earthkit.data.testing import main
 
diff --git a/tests/core/test_settings.py b/tests/core/test_settings.py
index 4f6e143..c284f85 100644
--- a/tests/core/test_settings.py
+++ b/tests/core/test_settings.py
@@ -15,6 +15,19 @@ from earthkit.data import settings
 from earthkit.data.core.temporary import temp_directory
 
 
+def read_settings_yaml(path="~/.earthkit-data/settings.yaml"):
+    try:
+        with open(path) as f:
+            import yaml
+
+            s = yaml.load(f, Loader=yaml.SafeLoader)
+            if not isinstance(s, dict):
+                s = {}
+            return s
+    except Exception:
+        return {}
+
+
 @pytest.mark.parametrize(
     "param,default_value,new_value",
     [
@@ -77,7 +90,9 @@ def test_settings_set_numbers(param, set_value, stored_value, raise_error):
 
 def test_settings_set_cache_numbers():
     with temp_directory() as tmpdir:
-        with settings.temporary("cache-directory", tmpdir):
+        with settings.temporary(
+            {"cache-policy": "user", "user-cache-directory": tmpdir}
+        ):
             data = [
                 ("maximum-cache-size", "1", 1, None),
                 ("maximum-cache-size", "1k", 1024, None),
@@ -102,6 +117,82 @@ def test_settings_set_cache_numbers():
                         settings.set(param, set_value)
 
 
+def test_settings_set_multi():
+    with settings.temporary():
+        settings.set("number-of-download-threads", 7)
+        assert settings.get("number-of-download-threads") == 7
+
+        settings.set({"number-of-download-threads": 2, "url-download-timeout": 21})
+        assert settings.get("number-of-download-threads") == 2
+        assert settings.get("url-download-timeout") == 21
+
+        settings.set(number_of_download_threads=3, url_download_timeout=11)
+        assert settings.get("number-of-download-threads") == 3
+        assert settings.get("url-download-timeout") == 11
+
+        with pytest.raises(KeyError):
+            settings.set({"number-of-download-threads": 2, "-invalid-": 21})
+
+        with pytest.raises(KeyError):
+            settings.set(number_of_download_threads=3, __invalid__=11)
+
+
+def test_settings_temporary_single():
+    with settings.temporary("number-of-download-threads", 7):
+        assert settings.get("number-of-download-threads") == 7
+
+    with settings.temporary({"number-of-download-threads": 7}):
+        assert settings.get("number-of-download-threads") == 7
+
+    with settings.temporary(number_of_download_threads=7):
+        assert settings.get("number-of-download-threads") == 7
+
+
+def test_settings_temporary_multi():
+    with settings.temporary(
+        {"number-of-download-threads": 2, "url-download-timeout": 21}
+    ):
+        assert settings.get("number-of-download-threads") == 2
+        assert settings.get("url-download-timeout") == 21
+
+    with settings.temporary(number_of_download_threads=3, url_download_timeout=11):
+        assert settings.get("number-of-download-threads") == 3
+        assert settings.get("url-download-timeout") == 11
+
+
+def test_settings_temporary_nested():
+    with settings.temporary("number-of-download-threads", 7):
+        assert settings.get("number-of-download-threads") == 7
+        with settings.temporary("number-of-download-threads", 10):
+            assert settings.get("number-of-download-threads") == 10
+        assert settings.get("number-of-download-threads") == 7
+
+
+@pytest.mark.parametrize("autosave", [True, False])
+def test_settings_temporary_autosave(autosave):
+    v_ori = settings.auto_save_settings
+    s = read_settings_yaml()
+    if s:
+        with settings.temporary():
+            settings.auto_save_settings = autosave
+            v = settings.get("number-of-download-threads")
+            settings.set("number-of-download-threads", v + 10)
+            assert s["number-of-download-threads"] == v
+        assert settings.auto_save_settings == autosave
+    settings.auto_save_settings = v_ori
+
+
+def test_settings_auto_save():
+    v_ori = settings.auto_save_settings
+    settings.auto_save_settings = False
+    s = read_settings_yaml()
+    if s:
+        v = settings.get("number-of-download-threads")
+        settings.set("number-of-download-threads", v + 10)
+        assert s["number-of-download-threads"] == v
+    settings.auto_save_settings = v_ori
+
+
 if __name__ == "__main__":
     from earthkit.data.testing import main
 
diff --git a/tests/documentation/test_notebooks.py b/tests/documentation/test_notebooks.py
index 3ef29f1..fcaba58 100644
--- a/tests/documentation/test_notebooks.py
+++ b/tests/documentation/test_notebooks.py
@@ -22,7 +22,7 @@ from earthkit.data.testing import MISSING, earthkit_file
 
 EXAMPLES = earthkit_file("docs", "examples")
 
-SKIP = ["fdb.ipynb", "mars.ipynb", "cds.ipynb"]
+SKIP = ["fdb.ipynb", "mars.ipynb", "cds.ipynb", "ads.ipynb"]
 
 
 def notebooks_list():
diff --git a/tests/sources/test_ads.py b/tests/sources/test_ads.py
new file mode 100644
index 0000000..6c5ad02
--- /dev/null
+++ b/tests/sources/test_ads.py
@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+
+# (C) Copyright 2020 ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+#
+
+import os
+
+import pytest
+
+from earthkit.data import from_source
+
+NO_ADS = not os.path.exists(os.path.expanduser("~/.adsapirc"))
+
+
+@pytest.mark.long_test
+@pytest.mark.download
+@pytest.mark.skipif(NO_ADS, reason="No access to ADS")
+def test_ads_grib_1():
+    s = from_source(
+        "ads",
+        "cams-global-reanalysis-eac4",
+        variable=["particulate_matter_10um", "particulate_matter_1um"],
+        area=[50, -50, 20, 50],
+        date="2012-12-12",
+        time="12:00",
+    )
+    assert len(s) == 2
+
+
+@pytest.mark.long_test
+@pytest.mark.download
+@pytest.mark.skipif(NO_ADS, reason="No access to ADS")
+def test_ads_grib_2():
+    s = from_source(
+        "ads",
+        "cams-global-reanalysis-eac4",
+        variable=["particulate_matter_10um", "particulate_matter_1um"],
+        area=[50, -50, 20, 50],
+        date="2012-12-12",
+        time="12:00",
+        split_on="variable",
+    )
+    assert len(s) == 2
+
+
+@pytest.mark.long_test
+@pytest.mark.download
+@pytest.mark.skipif(NO_ADS, reason="No access to ADS")
+def test_ads_grib_3():
+    s = from_source(
+        "ads",
+        "cams-global-reanalysis-eac4",
+        variable=["particulate_matter_10um", "particulate_matter_1um"],
+        area=[50, -50, 20, 50],
+        date="2012-12-12/to/2012-12-15",
+        time="12:00",
+    )
+    assert len(s) == 8
+
+
+@pytest.mark.long_test
+@pytest.mark.download
+@pytest.mark.skipif(NO_ADS, reason="No access to ADS")
+def test_ads_netcdf():
+    s = from_source(
+        "ads",
+        "cams-global-reanalysis-eac4",
+        variable=["particulate_matter_10um", "particulate_matter_1um"],
+        area=[50, -50, 20, 50],
+        date="2012-12-12",
+        time="12:00",
+        format="netcdf",
+    )
+    assert len(s) == 2
+
+
+if __name__ == "__main__":
+    from earthkit.data.testing import main
+
+    main(__file__)
diff --git a/tests/sources/test_url.py b/tests/sources/test_url.py
index 38e66eb..0319dde 100644
--- a/tests/sources/test_url.py
+++ b/tests/sources/test_url.py
@@ -45,7 +45,7 @@ def test_url_source_check_out_of_date():
 
     with temp_directory() as tmpdir:
         with settings.temporary():
-            settings.set("cache-directory", tmpdir)
+            settings.set("user-cache-directory", tmpdir)
             load()
 
             settings.set("check-out-of-date-urls", False)
diff --git a/tests/utils/__init__.py b/tests/utils/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/tests/utils/dummy_module.py b/tests/utils/dummy_module.py
new file mode 100644
index 0000000..6165eb1
--- /dev/null
+++ b/tests/utils/dummy_module.py
@@ -0,0 +1,23 @@
+import typing as T
+
+import numpy as np
+import pandas as pd
+import xarray as xr
+
+dummy_attribute_1 = 1
+dummy_attribute_2 = 2
+
+XR_TYPES = (xr.Dataset, xr.DataArray, xr.Variable)
+
+
+def numpy_mean(
+    a: T.Union[
+        np.ndarray, xr.Dataset, xr.DataArray, xr.Variable, pd.DataFrame, pd.Series
+    ],
+    **kwargs
+):
+    return np.mean(a, **kwargs)
+
+
+def xarray_ones_like(dataarray: T.Union[xr.DataArray, xr.Dataset], **kwargs):
+    return xr.ones_like(dataarray, **kwargs)
diff --git a/tests/utils/test_module_inputs_wrapper.py b/tests/utils/test_module_inputs_wrapper.py
new file mode 100644
index 0000000..3df34b8
--- /dev/null
+++ b/tests/utils/test_module_inputs_wrapper.py
@@ -0,0 +1,170 @@
+# (C) Copyright 2020 ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+#
+
+import numpy as np
+import pandas as pd
+import xarray as xr
+
+from earthkit.data import from_object, from_source
+from earthkit.data.readers import Reader
+from earthkit.data.utils import module_inputs_wrapper
+
+from . import dummy_module
+from .dummy_module import XR_TYPES
+
+TEST_NP = np.arange(10)
+TEST_NP2 = np.arange(10)
+
+TEST_DF = pd.DataFrame({"index": TEST_NP, "data": TEST_NP2}).set_index("index")
+
+TEST_DA = xr.DataArray(TEST_NP, name="test")
+TEST_DA2 = xr.DataArray(TEST_NP2, name="test2")
+TEST_DS = TEST_DA.to_dataset()
+TEST_DS["test2"] = TEST_DA2
+
+EK_GRIB_READER = from_source("file", "tests/data/test_single.grib")
+EK_XARRAY_WRAPPER = from_object(TEST_DS)
+EK_NUMPY_WRAPPER = from_object(TEST_NP)
+
+WRAPPED_XR_ONES_LIKE = module_inputs_wrapper.transform_function_inputs(
+    xr.ones_like, kwarg_types={"other": XR_TYPES}
+)
+
+WRAPPED_XR_ONES_LIKE_TYPE_SETTING = module_inputs_wrapper.transform_function_inputs(
+    dummy_module.xarray_ones_like,
+)
+
+WRAPPED_NP_MEAN = module_inputs_wrapper.transform_function_inputs(
+    np.mean,
+    kwarg_types={"a": np.ndarray},
+    convert_types=(
+        Reader
+    ),  # Only convert Earthkit.data.Reader (np.mean can handle xarray and pandas)
+)
+
+WRAPPED_NP_MEAN_TYPE_SETTING = module_inputs_wrapper.transform_function_inputs(
+    dummy_module.numpy_mean,
+)
+
+WRAPPED_DUMMY_MODULE = module_inputs_wrapper.transform_module_inputs(
+    dummy_module,
+)
+
+
+def test_transform_function_inputs_reader_to_xarray():
+    # Check EK GribReader object
+    ek_reader_result = WRAPPED_XR_ONES_LIKE(EK_GRIB_READER)
+    # Will return a DataSet becuase that is first value in kwarg_types
+    assert isinstance(ek_reader_result, xr.Dataset)
+    assert ek_reader_result.equals(xr.ones_like(EK_GRIB_READER.to_xarray()))
+
+
+def test_transform_function_inputs_reader_to_xarray_typesetting():
+    # Check EK GribReader object
+    ek_reader_result = WRAPPED_XR_ONES_LIKE_TYPE_SETTING(EK_GRIB_READER)
+    # Will return a dataarray because that is first value in type-set Union
+    assert isinstance(ek_reader_result, xr.DataArray)
+    assert ek_reader_result.equals(xr.ones_like(EK_GRIB_READER.to_xarray().t2m))
+
+
+def test_transform_module_inputs_reader_to_xarray():
+    # Check EK GribReader object
+    ek_reader_result = WRAPPED_DUMMY_MODULE.xarray_ones_like(EK_GRIB_READER)
+    # Data array because type-setting of function has dataarray first
+    assert isinstance(ek_reader_result, xr.DataArray)
+    assert ek_reader_result.equals(xr.ones_like(EK_GRIB_READER.to_xarray()).t2m)
+
+
+def test_transform_function_inputs_wrapper_to_xarray():
+    # EK XarrayWrapper object
+    ek_wrapper_result = WRAPPED_XR_ONES_LIKE(EK_XARRAY_WRAPPER)
+    assert isinstance(ek_wrapper_result, xr.Dataset)
+    assert ek_wrapper_result.equals(xr.ones_like(EK_XARRAY_WRAPPER.data))
+    # EK NumpyWrapper object
+    ek_wrapper_result = WRAPPED_XR_ONES_LIKE(EK_NUMPY_WRAPPER)
+    assert isinstance(ek_wrapper_result, xr.DataArray)
+    assert ek_wrapper_result.equals(xr.ones_like(TEST_DA))
+
+
+def test_transform_module_inputs_wrapper_to_xarray():
+    # EK XarrayWrapper object
+    ek_wrapper_result = WRAPPED_DUMMY_MODULE.xarray_ones_like(EK_XARRAY_WRAPPER)
+    # Will return a dataarray because that is first value in type-set Union
+    assert isinstance(ek_wrapper_result, xr.DataArray)
+    assert ek_wrapper_result.equals(xr.ones_like(EK_XARRAY_WRAPPER.data.test))
+    # EK NumpyWrapper object
+    ek_wrapper_result = WRAPPED_DUMMY_MODULE.xarray_ones_like(EK_NUMPY_WRAPPER)
+    assert isinstance(ek_wrapper_result, xr.DataArray)
+    assert ek_wrapper_result.equals(xr.ones_like(TEST_DA))
+
+
+def test_transform_function_inputs_reader_to_numpy():
+    # Test with Earthkit.data GribReader object
+    assert WRAPPED_NP_MEAN(EK_GRIB_READER) == np.mean(EK_GRIB_READER.to_numpy())
+    assert isinstance(WRAPPED_NP_MEAN(EK_GRIB_READER), np.float64)
+
+
+def test_transform_function_inputs_reader_to_numpy_typesetting():
+    # Test with Earthkit.data GribReader object
+    result = WRAPPED_NP_MEAN_TYPE_SETTING(EK_GRIB_READER)
+    assert result == np.mean(EK_GRIB_READER.to_numpy())
+    assert isinstance(result, np.float64)
+
+
+def test_transform_module_inputs_reader_to_numpy():
+    # Test with Earthkit.data GribReader object
+    result = WRAPPED_DUMMY_MODULE.numpy_mean(EK_GRIB_READER)
+    assert result == np.mean(EK_GRIB_READER.to_numpy())
+    assert isinstance(result, np.float64)
+
+
+def test_transform_function_inputs_wrapper_to_numpy():
+    # Test with Earthkit.data XarrayWrapper object
+    ek_object_result = WRAPPED_NP_MEAN(EK_XARRAY_WRAPPER)
+    assert ek_object_result == np.mean(TEST_DS)
+    # assert isinstance(ek_object_result, type(EK_XARRAY_WRAPPER.data))
+
+    # Test with Earthkit.data NumpyWrapper object
+    ek_object_result = WRAPPED_NP_MEAN(EK_NUMPY_WRAPPER)
+    assert ek_object_result == np.mean(TEST_NP)
+    assert isinstance(ek_object_result, np.float64)
+
+
+def test_transform_module_inputs_wrapper_to_numpy():
+    # Test with Earthkit.data XarrayWrapper object
+    ek_object_result = WRAPPED_DUMMY_MODULE.numpy_mean(EK_XARRAY_WRAPPER)
+    assert ek_object_result == np.mean(TEST_DS)
+    # TODO: Without definition of convert_types, WRAPPERS are converted to first arguement
+    #       This could be addressed in future versions
+    # assert isinstance(ek_object_result, type(EK_XARRAY_WRAPPER.data))
+
+    # Test with Earthkit.data NumpyWrapper object
+    ek_object_result = WRAPPED_DUMMY_MODULE.numpy_mean(EK_NUMPY_WRAPPER)
+    assert ek_object_result == np.mean(TEST_NP)
+    assert isinstance(ek_object_result, np.float64)
+
+
+def test_transform_function_inputs_xarray_to_numpy():
+    # Test with xarray.DataArray object
+    assert WRAPPED_NP_MEAN(TEST_DA) == np.mean(TEST_NP)
+    assert WRAPPED_NP_MEAN(TEST_DA) == np.mean(TEST_DA)
+    assert isinstance(WRAPPED_NP_MEAN(TEST_DA), xr.DataArray)
+
+
+def test_transform_function_inputs_pandas_to_numpy():
+    # Test with pandas.DataFrame object, axis=0 is to preserve the pandas.DataFrame, this is pandas syntax
+    result = WRAPPED_NP_MEAN(TEST_DF, axis=0)
+    assert result.equals(np.mean(TEST_DF, axis=0))
+    assert isinstance(result, pd.core.series.Series)
+
+    # # Test without axis declartion, pandas>2 returns a scalar mean of the dataframe
+    # # Omitted as breaking change in pandas makes tests unstable
+    # result = WRAPPED_NP_MEAN(TEST_DF)
+    # assert result == np.mean(TEST_DF)
+    # assert isinstance(result, np.float64)