Repository: CS-SI/eodag (python)

Issue:
Capella Bucket Download issue
**Bucket Issue**
Hi there, I'm trying to download Capella's Open data from 
[this stac index]](https://www.stacindex.org/catalogs/capella-space-open-data#/BrVjRL9LGRjg8ipuaKLhdHr1kD4sbvyd8QTYGwbMTTtkR4fzPd4yeiApxyrQkiq/LGziSG3zruu42mUQkBqrXkRPLdBZyfQ9WBqmakLW8v6Q2jtwGXLNgDPJyeVfFR6XkZHa8qL5E75MA6d8vzhnxgNLQVUuVFNrq?t=2).
But I got a "NoSuchKey" error from the "bucket_name" = "data"

I tried unsuccesfull workarounds... If someone got an idea, it's very welcome.

Thanks, and thanks a lot EOdag devs!
Ari

**Code To Reproduce**

```py
from eodag import setup_logging
setup_logging(verbose=3)

from eodag.api.core import EODataAccessGateway

# Create an EODAG custom STAC provider
dag = EODataAccessGateway()

# Add the custom STAC provider + output + login an password
catalog_path = "https://capella-open-data.s3.us-west-2.amazonaws.com/stac/capella-open-data-by-product-type/capella-open-data-slc/collection.json"
# catalog_path = 'https://capella-open-data.s3.us-west-2.amazonaws.com/stac/catalog.json'
base_uri = "https://capella-open-data.s3.us-west-2.amazonaws.com"
outputs_prefix = r"D:\Radar\data\EODAG"
aws_access_key_id = "change_me"
aws_secret_access_key = "change_me"


dag.update_providers_config("""
capella:
    search:
        type: StaticStacSearch
        api_endpoint: %s
    products:
        GENERIC_PRODUCT_TYPE:
            productType: '{productType}'
    download:
        type: AwsDownload
        base_uri: %s
        flatten_top_dirs: True
        outputs_prefix: %s
    auth:
        type: AwsAuth
        credentials:
            aws_access_key_id: %s
            aws_secret_access_key: %s
""" % (catalog_path, base_uri, outputs_prefix, aws_access_key_id, aws_secret_access_key ))

# # Set the custom STAC provider as preferred
dag.set_preferred_provider("capella")

# Query every product from inside the catalog
# all_products, _ = dag.search()

capella_prod, _ = dag.search(start="2021-12-01", end="2021-12-15", **{"sar:product_type": "SLC", "sensorMode": "spotlight"})  # , locations=dict(country="BRA"), start="2021-01-01", end="2020-05-30", items_per_page=50)

# [(key, asset["href"]) for key, asset in capella_prod[0].assets.items()]

paths = dag.download_all(capella_prod)

```

**TroubleShoot**
```py
2022-10-07 16:47:56,830 eodag.config                     [INFO    ] (config           ) Loading user configuration from: C:\Users\ajeannin\.config\eodag\eodag.yml
2022-10-07 16:47:56,923 eodag.core                       [INFO    ] (core             ) usgs: provider needing auth for search has been pruned because no crendentials could be found
2022-10-07 16:47:56,923 eodag.core                       [INFO    ] (core             ) aws_eos: provider needing auth for search has been pruned because no crendentials could be found
2022-10-07 16:47:57,010 eodag.core                       [DEBUG   ] (core             ) Opening product types index in C:\Users\ajeannin\.config\eodag\.index
2022-10-07 16:47:57,020 eodag.core                       [INFO    ] (core             ) Locations configuration loaded from C:\Users\ajeannin\.config\eodag\locations.yml
2022-10-07 16:47:57,022 eodag.config                     [INFO    ] (config           ) capella: unknown provider found in user conf, trying to use provided configuration
2022-10-07 16:47:57,110 eodag.core                       [INFO    ] (core             ) No product type could be guessed with provided arguments
2022-10-07 16:47:57,686 eodag.core                       [INFO    ] (core             ) Searching product type 'None' on provider: capella
2022-10-07 16:47:57,686 eodag.core                       [DEBUG   ] (core             ) Using plugin class for search: StaticStacSearch
2022-10-07 16:47:58,682 eodag.utils.stac_reader          [DEBUG   ] (stac_reader      ) Fetching 125 items
2022-10-07 16:47:58,683 eodag.utils.stac_reader          [DEBUG   ] (stac_reader      ) read_local_json is not the right STAC opener
2022-10-07 16:48:00,570 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Getting genric provider product type definition parameters for None
2022-10-07 16:48:00,571 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Building the query string that will be used for search
2022-10-07 16:48:00,571 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Retrieving queryable metadata from metadata_mapping
2022-10-07 16:48:00,609 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page URL could not be collected
2022-10-07 16:48:00,626 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page Query-object could not be collected
2022-10-07 16:48:00,643 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page merge could not be collected
2022-10-07 16:48:00,643 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Adapting 125 plugin results to eodag product representation
2022-10-07 16:48:02,877 eodag.plugins.crunch.filter_date [DEBUG   ] (filter_date      ) Start filtering by date
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_date [INFO    ] (filter_date      ) Finished filtering products. 4 resulting products
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_overlap [DEBUG   ] (filter_overlap   ) Start filtering for overlapping products
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_overlap [WARNING ] (filter_overlap   ) geometry not found in cruncher arguments, filtering disabled.
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_property [DEBUG   ] (filter_property  ) Start filtering for products matching operator.eq(product.properties['sar:product_type'], SLC)
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_property [INFO    ] (filter_property  ) Finished filtering products. 4 resulting products
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_property [DEBUG   ] (filter_property  ) Start filtering for products matching operator.eq(product.properties['sensorMode'], spotlight)
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_property [INFO    ] (filter_property  ) Finished filtering products. 2 resulting products
2022-10-07 16:48:02,898 eodag.core                       [INFO    ] (core             ) Found 2 result(s) on provider 'capella'
2022-10-07 16:48:02,898 eodag.core                       [INFO    ] (core             ) Downloading 2 products
Downloaded products:   0%|          | 0/2 [00:00<?, ?product/s]
0.00B [00:00, ?B/s]
CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646: 0.00B [00:00, ?B/s]2022-10-07 16:48:02,900 eodag.plugins.download.base      [INFO    ] (base             ) Download url: https://capella-open-data.s3.us-west-2.amazonaws.com/collections/capella-open-data-spotlight/items/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646
2022-10-07 16:48:03,847 eodag.plugins.download.aws       [WARNING ] (aws              ) Unexpected error: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
2022-10-07 16:48:03,847 eodag.plugins.download.aws       [WARNING ] (aws              ) Skipping data//2021/12/14/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646.tif
2022-10-07 16:48:04,678 eodag.plugins.download.aws       [WARNING ] (aws              ) Unexpected error: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
2022-10-07 16:48:04,678 eodag.plugins.download.aws       [WARNING ] (aws              ) Skipping data//2021/12/14/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646_extended.json
2022-10-07 16:48:05,496 eodag.plugins.download.aws       [WARNING ] (aws              ) Unexpected error: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
2022-10-07 16:48:05,496 eodag.plugins.download.aws       [WARNING ] (aws              ) Skipping data//2021/12/14/CAPELLA_C03_SP_GEO_HH_20211214232633_20211214232656/CAPELLA_C03_SP_GEO_HH_20211214232633_20211214232656_preview.tif
2022-10-07 16:48:06,301 eodag.plugins.download.aws       [WARNING ] (aws              ) Unexpected error: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
2022-10-07 16:48:06,301 eodag.plugins.download.aws       [WARNING ] (aws              ) Skipping data//2021/12/14/CAPELLA_C03_SP_GEO_HH_20211214232633_20211214232656/CAPELLA_C03_SP_GEO_HH_20211214232633_20211214232656_thumb.png
2022-10-07 16:48:06,302 eodag.plugins.download.base      [ERROR   ] (base             ) Stopped because of credentials problems with provider capella
Traceback (most recent call last):
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\base.py", line 456, in download_all
    product.download(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\api\product\_product.py", line 288, in download
    fs_path = self.downloader.download(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\aws.py", line 339, in download
    raise AuthenticationError(", ".join(auth_error_messages))
eodag.utils.exceptions.AuthenticationError: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
Downloaded products:   0%|          | 0/2 [00:03<?, ?product/s]
Traceback (most recent call last):
  File "D:\PythonCode\radar_dev\radar_dev\utils.py", line 48, in <module>
    paths = dag.download_all(capella_prod)
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\api\core.py", line 1288, in download_all
    paths = download_plugin.download_all(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\aws.py", line 920, in download_all
    return super(AwsDownload, self).download_all(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\base.py", line 456, in download_all
    product.download(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\api\product\_product.py", line 288, in download
    fs_path = self.downloader.download(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\aws.py", line 339, in download
    raise AuthenticationError(", ".join(auth_error_messages))
eodag.utils.exceptions.AuthenticationError: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646: 0.00B [00:03, ?B/s]

Process finished with exit code 1
```

**Environment:**
 - Conda version: 22.9.0
 - Python version: 3.7
 - EODAG version: `eodag version` tells me 
 ```py
 (radar_dev) λ eodag version                                                                                                                                                                                                                                                                          
eodag (Earth Observation Data Access Gateway): version 0.0.0
```
But I think I got the 2.5.0 version as it is the latest on Conda env (should I open a new issue ?)




Pull request:
fix: aws bucket extraction
- Refactors and updates `get_bucket_name_and_prefix()`
- updates `mundi` configuration with `bucket_path_level = 0` and up-to-date search endpoint

fixes #519 

Patch:
diff --git a/eodag/plugins/download/aws.py b/eodag/plugins/download/aws.py
index e27ffe9c..e5d9d2a9 100644
--- a/eodag/plugins/download/aws.py
+++ b/eodag/plugins/download/aws.py
@@ -34,7 +34,12 @@ from eodag.api.product.metadata_mapping import (
     properties_from_xml,
 )
 from eodag.plugins.download.base import Download
-from eodag.utils import ProgressCallback, path_to_uri, rename_subfolder, urlparse
+from eodag.utils import (
+    ProgressCallback,
+    get_bucket_name_and_prefix,
+    path_to_uri,
+    rename_subfolder,
+)
 from eodag.utils.exceptions import AuthenticationError, DownloadError
 from eodag.utils.stac_reader import HTTP_REQ_TIMEOUT
 
@@ -601,22 +606,20 @@ class AwsDownload(Download):
         """
         if not url:
             url = product.location
-        bucket, prefix = None, None
-        # Assume the bucket is encoded into the product location as a URL or given as the 'default_bucket' config
-        # param
-        scheme, netloc, path, params, query, fragment = urlparse(url)
-        if not scheme or scheme == "s3":
+
+        bucket_path_level = getattr(self.config, "bucket_path_level", None)
+
+        bucket, prefix = get_bucket_name_and_prefix(
+            url=url, bucket_path_level=bucket_path_level
+        )
+
+        if bucket is None:
             bucket = (
-                netloc
-                if netloc
-                else getattr(self.config, "products", {})
+                getattr(self.config, "products", {})
                 .get(product.product_type, {})
                 .get("default_bucket", "")
             )
-            prefix = path.strip("/")
-        elif scheme in ("http", "https", "ftp"):
-            parts = path.split("/")
-            bucket, prefix = parts[1], "/{}".format("/".join(parts[2:]))
+
         return bucket, prefix
 
     def check_manifest_file_list(self, product_path):
diff --git a/eodag/plugins/download/s3rest.py b/eodag/plugins/download/s3rest.py
index db92da7a..fa2d0f65 100644
--- a/eodag/plugins/download/s3rest.py
+++ b/eodag/plugins/download/s3rest.py
@@ -27,10 +27,15 @@ import requests
 from requests import RequestException
 
 from eodag.api.product.metadata_mapping import OFFLINE_STATUS
-from eodag.plugins.download.aws import AwsDownload
-from eodag.plugins.download.base import DEFAULT_STREAM_REQUESTS_TIMEOUT
+from eodag.plugins.download.base import DEFAULT_STREAM_REQUESTS_TIMEOUT, Download
 from eodag.plugins.download.http import HTTPDownload
-from eodag.utils import ProgressCallback, path_to_uri, unquote, urljoin
+from eodag.utils import (
+    ProgressCallback,
+    get_bucket_name_and_prefix,
+    path_to_uri,
+    unquote,
+    urljoin,
+)
 from eodag.utils.exceptions import (
     AuthenticationError,
     DownloadError,
@@ -42,7 +47,7 @@ from eodag.utils.stac_reader import HTTP_REQ_TIMEOUT
 logger = logging.getLogger("eodag.plugins.download.s3rest")
 
 
-class S3RestDownload(AwsDownload):
+class S3RestDownload(Download):
     """Http download on S3-like object storage location
     for example using Mundi REST API (free account)
     https://mundiwebservices.com/keystoneapi/uploads/documents/CWS-DATA-MUT-087-EN-Mundi_Download_v1.1.pdf#page=13
@@ -78,7 +83,9 @@ class S3RestDownload(AwsDownload):
             progress_callback = ProgressCallback(disable=True)
 
         # get bucket urls
-        bucket_name, prefix = self.get_bucket_name_and_prefix(product)
+        bucket_name, prefix = get_bucket_name_and_prefix(
+            url=product.location, bucket_path_level=self.config.bucket_path_level
+        )
 
         if (
             bucket_name is None
diff --git a/eodag/resources/providers.yml b/eodag/resources/providers.yml
index 0302a9b2..df79e7aa 100644
--- a/eodag/resources/providers.yml
+++ b/eodag/resources/providers.yml
@@ -1261,7 +1261,7 @@
   url: https://mundiwebservices.com/
   search: !plugin
     type: QueryStringSearch
-    api_endpoint: 'https://mundiwebservices.com/acdc/catalog/proxy/search/{collection}/opensearch'
+    api_endpoint: 'https://{collection}.browse.catalog.mundiwebservices.com/opensearch'
     need_auth: false
     result_type: 'xml'
     results_entry: '//ns:entry'
@@ -1390,6 +1390,7 @@
     base_uri: 'https://mundiwebservices.com/dp'
     extract: true
     auth_error_code: 401
+    bucket_path_level: 0
   auth: !plugin
     type: HTTPHeaderAuth
     headers:
diff --git a/eodag/utils/__init__.py b/eodag/utils/__init__.py
index de74e38b..40234376 100644
--- a/eodag/utils/__init__.py
+++ b/eodag/utils/__init__.py
@@ -1028,3 +1028,33 @@ def cached_parse(str_to_parse):
     :rtype: :class:`jsonpath_ng.JSONPath`
     """
     return parse(str_to_parse)
+
+
+def get_bucket_name_and_prefix(url=None, bucket_path_level=None):
+    """Extract bucket name and prefix from URL
+
+    :param url: (optional) URL to use as product.location
+    :type url: str
+    :param bucket_path_level: (optional) bucket location index in path.split('/')
+    :type bucket_path_level: int
+    :returns: bucket_name and prefix as str
+    :rtype: tuple
+    """
+    bucket, prefix = None, None
+
+    scheme, netloc, path, params, query, fragment = urlparse(url)
+    subdomain = netloc.split(".")[0]
+    path = path.strip("/")
+
+    if scheme and bucket_path_level is None:
+        bucket = subdomain
+        prefix = path
+    elif not scheme and bucket_path_level is None:
+        prefix = path
+    elif bucket_path_level is not None:
+        parts = path.split("/")
+        bucket, prefix = parts[bucket_path_level], "/".join(
+            parts[(bucket_path_level + 1) :]
+        )
+
+    return bucket, prefix


Test patch:
diff --git a/tests/context.py b/tests/context.py
index 1428ddf0..f23c149a 100644
--- a/tests/context.py
+++ b/tests/context.py
@@ -61,6 +61,7 @@ from eodag.plugins.download.http import HTTPDownload
 from eodag.plugins.manager import PluginManager
 from eodag.plugins.search.base import Search
 from eodag.utils import (
+    get_bucket_name_and_prefix,
     get_geometry_from_various,
     get_timestamp,
     makedirs,
diff --git a/tests/units/test_download_plugins.py b/tests/units/test_download_plugins.py
index 40839b23..3b3c480b 100644
--- a/tests/units/test_download_plugins.py
+++ b/tests/units/test_download_plugins.py
@@ -314,3 +314,40 @@ class TestDownloadPluginHttpRetry(BaseDownloadPluginTest):
             self.assertEqual(len(responses.calls), 1)
 
         run()
+
+
+class TestDownloadPluginAws(BaseDownloadPluginTest):
+    def setUp(self):
+        super(TestDownloadPluginAws, self).setUp()
+        self.product = EOProduct(
+            "astraea_eod",
+            dict(
+                geometry="POINT (0 0)",
+                title="dummy_product",
+                id="dummy",
+            ),
+            productType="S2_MSI_L2A",
+        )
+
+    def test_plugins_download_aws_get_bucket_prefix(self):
+        """AwsDownload.get_bucket_name_and_prefix() must extract bucket & prefix from location"""
+
+        plugin = self.get_download_plugin(self.product)
+        plugin.config.products["S2_MSI_L2A"]["default_bucket"] = "default_bucket"
+        self.product.properties["id"] = "someproduct"
+
+        self.product.location = (
+            self.product.remote_location
+        ) = "http://somebucket.somehost.com/path/to/some/product"
+        bucket, prefix = plugin.get_bucket_name_and_prefix(self.product)
+        self.assertEqual((bucket, prefix), ("somebucket", "path/to/some/product"))
+
+        plugin.config.bucket_path_level = 1
+        bucket, prefix = plugin.get_bucket_name_and_prefix(self.product)
+        self.assertEqual((bucket, prefix), ("to", "some/product"))
+        del plugin.config.bucket_path_level
+
+        bucket, prefix = plugin.get_bucket_name_and_prefix(
+            self.product, url="/somewhere/else"
+        )
+        self.assertEqual((bucket, prefix), ("default_bucket", "somewhere/else"))
diff --git a/tests/units/test_utils.py b/tests/units/test_utils.py
index f07a3c71..d5596270 100644
--- a/tests/units/test_utils.py
+++ b/tests/units/test_utils.py
@@ -25,6 +25,7 @@ from io import StringIO
 from tests.context import (
     DownloadedCallback,
     ProgressCallback,
+    get_bucket_name_and_prefix,
     get_timestamp,
     merge_mappings,
     path_to_uri,
@@ -162,3 +163,51 @@ class TestUtils(unittest.TestCase):
         mapping = {"keyA": True}
         merge_mappings(mapping, {"keya": "bar"})
         self.assertEqual(mapping, {"keyA": True})
+
+    def test_get_bucket_name_and_prefix(self):
+        """get_bucket_name_and_prefix must extract bucket and prefix from url"""
+        self.assertEqual(
+            get_bucket_name_and_prefix(
+                "s3://sentinel-s2-l1c/tiles/50/R/LR/2021/6/8/0/B02.jp2"
+            ),
+            ("sentinel-s2-l1c", "tiles/50/R/LR/2021/6/8/0/B02.jp2"),
+        )
+        self.assertEqual(
+            get_bucket_name_and_prefix(
+                "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/44/2022/10/S2B_20221011/B01.tif"
+            ),
+            (
+                "sentinel-cogs",
+                "sentinel-s2-l2a-cogs/44/2022/10/S2B_20221011/B01.tif",
+            ),
+        )
+        self.assertEqual(
+            get_bucket_name_and_prefix(
+                "https://obs.eu-de.otc.t-systems.com/s2-l1c-2021-q4/30/T/2021/11/14/S2A_MSIL1C_20211114T110321_N0301",
+                0,
+            ),
+            ("s2-l1c-2021-q4", "30/T/2021/11/14/S2A_MSIL1C_20211114T110321_N0301"),
+        )
+        self.assertEqual(
+            get_bucket_name_and_prefix(
+                "products/2021/1/5/S2A_MSIL1C_20210105T105441_N0209_R051_T30TYN_20210105T130129",
+            ),
+            (
+                None,
+                "products/2021/1/5/S2A_MSIL1C_20210105T105441_N0209_R051_T30TYN_20210105T130129",
+            ),
+        )
+        self.assertEqual(
+            get_bucket_name_and_prefix(
+                "bucket/path/to/product",
+                0,
+            ),
+            ("bucket", "path/to/product"),
+        )
+        self.assertEqual(
+            get_bucket_name_and_prefix(
+                "http://foo/bar/bucket/path/to/product",
+                1,
+            ),
+            ("bucket", "path/to/product"),
+        )