Repository: ecmwf/earthkit-data
Pull request #318 (2024-02-23 09:59:08)

Add source to load sample data
This PR adds the `from_sample` method to access data stored on the download server in folder `earthkit/samples`. The primary goal is to help users to quickly access data used for the examples in `earthkit-maps` and `earthkit-data`.

This is how it works:

```python
from earthkit.data import from_sample
ds = from_sample("storm_ophelia_wind_850.grib")
```

Alternatively we could introduce another source "sample" to implement this feature, e.g.:


```python
from earthkit.data import from_source
ds = from_source("sample", "storm_ophelia_wind_850.grib")
```


sandorkertesz: The feature is now implemented as a sample source:

![image](https://github.com/ecmwf/earthkit-data/assets/9033020/e5368d27-81ab-4fb6-9d79-b0d0bd2c27f9)


Patch:
diff --git a/docs/guide/sources.rst b/docs/guide/sources.rst
index f232d4c..2c579c0 100644
--- a/docs/guide/sources.rst
+++ b/docs/guide/sources.rst
@@ -32,6 +32,8 @@ We can get data from a given source by using :func:`from_source`:
       - read data from a URL
     * - :ref:`data-sources-url-pattern`
       - read data from a list of URLs created from a pattern
+    * - :ref:`data-sources-sample`
+      - read example data
     * - :ref:`data-sources-stream`
       - read data from a stream
     * - :ref:`data-sources-memory`
@@ -259,6 +261,32 @@ url-pattern
   ``url-pattern`` according to the **unpack** argument, similarly to what
   the source ``url`` does (see above the :ref:`data-sources-url` source).
 
+
+
+.. _data-sources-sample:
+
+sample
+------
+
+.. py:function:: from_source("sample", name_or_path)
+  :noindex:
+
+  The ``sample`` source will download example data prepared for earthkit and store it in the :ref:`cache <caching>`. The supported data formats are the same as for the :ref:`file <data-sources-file>` data source above.
+
+  :param name_or_path: input file name(s) or relative path(s) to the root of the remote storage folder.
+  :type name_or_path: str, list, tuple
+
+  .. code-block:: python
+
+    >>> import earthkit.data
+    >>> ds = earthkit.data.from_source("sample", "storm_ophelia_wind_850.grib")
+    >>> ds.ls()
+      centre shortName    typeOfLevel  level  dataDate  dataTime stepRange dataType  number    gridType
+    0   ecmf         u  isobaricInhPa    850  20171016         0         0       an       0  regular_ll
+    1   ecmf         v  isobaricInhPa    850  20171016         0         0       an       0  regular_ll
+
+
+
 .. _data-sources-stream:
 
 stream
diff --git a/earthkit/data/sources/sample.py b/earthkit/data/sources/sample.py
new file mode 100644
index 0000000..aeafe99
--- /dev/null
+++ b/earthkit/data/sources/sample.py
@@ -0,0 +1,40 @@
+# (C) Copyright 2020 ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+#
+
+import logging
+
+from .file import FileSource
+
+LOG = logging.getLogger(__name__)
+
+
+REMOTE_SAMPLE_DATA_URL = "https://get.ecmwf.int/repository/earthkit/samples/"
+
+
+class Sample(FileSource):
+    def __init__(self, path, *args, **kwargs):
+        super().__init__()
+
+        self._paths = path
+        if isinstance(self._paths, str):
+            self._paths = [path]
+
+        self._kwargs = dict(**kwargs)
+
+    def mutate(self):
+        from earthkit.data.sources.url import Url
+
+        urls = []
+        for f_name in self._paths:
+            urls.append(REMOTE_SAMPLE_DATA_URL + f_name)
+
+        return Url(urls, **self._kwargs)
+
+
+source = Sample


Test patch:
diff --git a/tests/sources/test_samples.py b/tests/sources/test_samples.py
new file mode 100644
index 0000000..ce58412
--- /dev/null
+++ b/tests/sources/test_samples.py
@@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+
+# (C) Copyright 2020 ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+#
+
+from earthkit.data import from_source
+
+
+def test_sample_source_grib():
+    ds = from_source("sample", "storm_ophelia_wind_850.grib")
+    assert len(ds) == 2
+    assert ds.metadata(["param", "level"]) == [["u", 850], ["v", 850]]
+
+
+if __name__ == "__main__":
+    from earthkit.data.testing import main
+
+    main(__file__)