Repository: dlr-eoc/ukis-pysat (language: python)

Issue:
Develop our own STAC-API search
**Roadmap to move `ukis_pysat.data` to STAC**

- [x] Develop our own STAC-API search class (inspired by [sat-search](https://github.com/sat-utils/sat-search/blob/master/satsearch/search.py))

_Originally posted by @MWieland in https://github.com/dlr-eoc/ukis-pysat/issues/98#issuecomment-760089477_

We have to do our own because it has to be compatible with PySTAC for its IO capabilites.


Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 267e940..de3a634 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -15,6 +15,9 @@ something was changed
 
 [master]  (2020-**-**)
 ----------------------
+Added
+*****
+- ``data``: new stac api #101
 
 [0.7.0]  (2020-10-30)
 ---------------------
diff --git a/README.md b/README.md
index 5d15e03..391ae6e 100644
--- a/README.md
+++ b/README.md
@@ -12,15 +12,15 @@
 The UKIS-pysat package provides generic classes and functions to query, access and process multi-spectral and SAR satellite images.
 
 ### data
-Download satellites data from different sources (currently Earth Explorer, SciHub, local directory), deal with and structure metadata.
+Download satellites data from different sources (currently Earth Explorer, SciHub, STAC), deal with and structure metadata.
 
 
 ### file
-Work with you local satellite data files and read information out of file names and metadata files. Currently focusing on Sentinel-1.
+Work with you local satellite data files and read information out of file names and metadata files. Currently, focusing on Sentinel-1.
 
 
 ### raster
-Reading satellite data and performing simple, but cumbersome tasks. This is just a layer on top of [rasterio](https://github.com/mapbox/rasterio) for stuff we often need. It can very well be that using *rasterio* direclty is often the better choice.
+Reading satellite data and performing simple, but cumbersome tasks. This is just a layer on top of [rasterio](https://github.com/mapbox/rasterio) for stuff we often need. It can very well be that using *rasterio* directly is often the better choice.
 
 
 Read the documentation for more details: [https://ukis-pysat.readthedocs.io](https://ukis-pysat.readthedocs.io/en/latest/).
diff --git a/docs/api/index.rst b/docs/api/index.rst
index bbcf7b2..efa4bc9 100644
--- a/docs/api/index.rst
+++ b/docs/api/index.rst
@@ -4,6 +4,7 @@ Python API Reference
 .. toctree::
 
    data
+   stacapi
    file
    raster
 
diff --git a/docs/api/stacapi.rst b/docs/api/stacapi.rst
new file mode 100644
index 0000000..5486a6f
--- /dev/null
+++ b/docs/api/stacapi.rst
@@ -0,0 +1,7 @@
+ukis_pysat.\stacapi
+====================
+
+.. automodule:: ukis_pysat.stacapi
+  :members:
+  :undoc-members:
+  :show-inheritance:
\ No newline at end of file
diff --git a/docs/index.rst b/docs/index.rst
index f0e152a..56c7fba 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -8,6 +8,19 @@ Welcome to UKIS-pysat's documentation!
 
 The UKIS-pysat package provides generic classes and functions to query, access and process multi-spectral and SAR satellite images.
 
+data
+____
+Download satellites data from different sources (currently Earth Explorer, SciHub, STAC), deal with and structure metadata.
+
+file
+____
+Work with you local satellite data files and read information out of file names and metadata files. Currently, focusing on Sentinel-1.
+
+raster
+______
+Reading satellite data and performing simple, but cumbersome tasks. This is just a layer on top of `rasterio <https://github.com/mapbox/rasterio>`__ for stuff we often need. It can very well be that using *rasterio* directly is often the better choice.
+
+
 .. code-block:: python
 
   from ukis_pysat.data import Source
diff --git a/docs/installation.rst b/docs/installation.rst
index 9562107..0b825a0 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -35,9 +35,7 @@ To run the tests set the Environment variables and then:
 
     git clone https://github.com/dlr-eoc/ukis-pysat
     cd ukis-pysat
-    conda env create -f environment.yml
-    conda activate ukis_pysat
-    export
+    pip install -e .[dev]
     python -m unittest discover tests
 
 If you set the environment variables with the credentials to the hubs, you can uncomment `@unittest.skip()` for these tests.
\ No newline at end of file
diff --git a/requirements.txt b/requirements.txt
index 1d1b96e..9a4307b 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -9,6 +9,7 @@ pylandsat==0.4
 rio_toa==0.3.0
 pyproj~=2.6.1.post1
 pyfields~=1.6.0
+pystac>=0.5.4
 
 fiona~=1.8.11
 
diff --git a/ukis_pysat/_stacapiio.py b/ukis_pysat/_stacapiio.py
new file mode 100644
index 0000000..158033c
--- /dev/null
+++ b/ukis_pysat/_stacapiio.py
@@ -0,0 +1,27 @@
+from pystac import Item
+
+
+class StacApiIo:
+    def default_build_url_method(feature):
+        return Item.from_dict(feature)
+
+    def on_premise_s3_url_method(feature, root_bucket="dem"):
+        """the href is build like /collections/*collection_prefix*/items/*item_prefix*
+
+        At some environments at DLR we will need to give back the href according to this method.
+        """
+        item = Item.from_dict(feature)
+        href = item.get_self_href()
+        stripped_href = href.replace(r"collections/", "").replace(r"items/", "")
+
+        return Item.from_file(f"s3://{root_bucket}{stripped_href}/{item.id}.json")
+
+    url_method = default_build_url_method
+    """Like in PySTAC you can replace the parse_url_method to expand the ability to read different file systems.
+    For example, a client of the library might replace this class  member in it's own __init__.py with a method that 
+    can read from another cloud storage.
+    """
+
+    @classmethod
+    def build_url(cls, feature):
+        return cls.url_method(feature)
diff --git a/ukis_pysat/stacapi.py b/ukis_pysat/stacapi.py
new file mode 100644
index 0000000..1e9150d
--- /dev/null
+++ b/ukis_pysat/stacapi.py
@@ -0,0 +1,92 @@
+import os
+from urllib.parse import urljoin
+
+try:
+    import requests
+    from pystac import Item, Collection
+except ImportError as e:
+    msg = (
+        "ukis_pysat.data dependencies are not installed.\n\n"
+        "Please pip install as follows:\n\n"
+        "  python -m pip install ukis-pysat[data] --upgrade"
+    )
+    raise ImportError(str(e) + "\n\n" + msg)
+
+
+import ukis_pysat._stacapiio
+
+
+class StacApiError(Exception):
+    pass
+
+
+class StacApi:
+    # TODO be aware of https://github.com/azavea/franklin/issues/471
+    def __init__(self, url=os.getenv("STAC_API_URL", None)):
+        """API to query STAC as part of ukis-pysat.data
+        :param url: STAC Server endpoint, reads from STAC_API_URL environment variable by default
+        """
+        if url is None:
+            raise StacApiError("URL not provided, pass into StacApi or define STAC_API_URL environment variable")
+        self.url = url.rstrip("/") + "/"
+
+    def _handle_query(self, url=None, headers=None, **kwargs):
+        url = url or urljoin(self.url, "search")
+        response = self._query(url, kwargs=kwargs, headers=headers)
+        if response.status_code != 200:
+            raise StacApiError(response.text)
+        return response.json()
+
+    @staticmethod
+    def _query(url, kwargs, headers):
+        if "intersects" in kwargs:  # TODO intersects will be deprecated with v1.0.0-beta.2 and replaced with OGC CQL
+            return requests.post(url, json=kwargs, headers=headers)
+        else:
+            return requests.get(url, kwargs, headers=headers)
+
+    def count(self, headers=None, **kwargs):
+        """lightweight query to get count of results to expect
+        :param headers: headers (optional)
+        :param kwargs: search parameters (optional)
+        :returns number of found results"""
+        res = self._handle_query(headers=headers, **kwargs)
+        return res["context"]["matched"]
+
+    def get_items(self, limit=100, headers=None, **kwargs):
+        """get items or single item
+        :param limit: max number of items returned (default 100)
+        :param headers: headers (optional)
+        :param kwargs: search parameters (optional) See: https://github.com/radiantearth/stac-api-spec/tree/master/item-search#query-parameter-table
+        :returns list with pystac.items"""
+        next_page = urljoin(self.url, "search")
+        limit = kwargs.get("limit", limit)
+        items = []
+
+        while next_page:
+            res = self._handle_query(url=next_page, headers=headers, **kwargs)
+
+            if not res["links"]:
+                next_page = None
+            else:
+                next_page = res["links"][0]["href"] if res["links"][0]["rel"] == "next" else None
+
+            for f in res["features"]:
+                if len(items) == limit:
+                    next_page = None
+                    break
+                """Instead of reading Items from_dict we rebuild the URLs so that they fit to the file system in use"""
+                items.append(ukis_pysat._stacapiio.StacApiIo.build_url(f))
+
+        return items
+
+    def get_collections(self, collection_id=None, headers=None, **kwargs):
+        """get all collections or get collection by ID
+        :param collection_id: ID of collection (optional)
+        :param headers: headers (optional)
+        :param kwargs: search parameters (optional)
+        :returns list with pystac.collections"""
+        url = urljoin(self.url, f"collections/{collection_id}" if collection_id else "collections")
+        res = self._handle_query(url=url, headers=headers, **kwargs)
+        if isinstance(res, dict):
+            res = res.get("collections", [res])
+        return [Collection.from_dict(c) for c in res]


Test patch:
diff --git a/tests/test_stacapi.py b/tests/test_stacapi.py
new file mode 100644
index 0000000..3ac2005
--- /dev/null
+++ b/tests/test_stacapi.py
@@ -0,0 +1,55 @@
+import unittest
+
+import geojson
+from shapely import wkt
+
+from ukis_pysat.stacapi import StacApi, StacApiError
+
+
+class StacApiTest(unittest.TestCase):
+    def setUp(self):
+        self.url = "https://earth-search.aws.element84.com/v0/"
+        self.api = StacApi(url=self.url)
+
+        self.aoi = geojson.Feature(
+            geometry=wkt.loads(r"POLYGON((11.00 48.00, 11.05 48.00,11.05 48.05, 11.00 48.05, 11.00 48.00))"),
+            properties={},
+        ).geometry
+
+    def test_init(self):
+        self.assertEqual(self.api.url, self.url)
+
+    def test_no_url(self):
+        with self.assertRaises(StacApiError):
+            StacApi()
+
+    def test_collections(self):
+        collections = self.api.get_collections()
+        self.assertEqual(len(collections), 4)
+
+    def test_get_collections(self):
+        collection = self.api.get_collections(collection_id=r"sentinel-s2-l2a")
+        self.assertEqual(collection[0].id, r"sentinel-s2-l2a")
+
+    def test_count(self):
+        cnt = self.api.count(collection=r"sentinel-s2-l2a", datetime=r"2019-01-01T00:00:00Z/2019-01-01T23:59:59Z")
+        self.assertEqual(cnt, 25030)
+
+    @unittest.skip  # TODO does not work because endpoints expects quotes around ID
+    def test_get_item(self):
+        item = self.api.get_items(collection="sentinel-s2-l2a", ids=["S2A_35VLG_20210114_0_L2A"], limit=1)
+        self.assertEqual(item.id, "S2A_35VLG_20210114_0_L2A")
+
+    def test_get_item_intersects(self):
+        cnt = self.api.count(
+            collection="sentinel-s2-l2a", intersects=self.aoi, datetime=r"2020-04-01T00:00:00Z/2020-04-01T23:59:59Z"
+        )
+        self.assertEqual(3, cnt)
+
+    def test_get_items_limit(self, limit=31):
+        items = self.api.get_items(collection="sentinel-s2-l2a", intersects=self.aoi, limit=limit)
+        self.assertEqual(limit, len(items))
+
+
+if __name__ == "__main__":
+    unittest.main()