Repository: CS-SI/eodag
Pull request #493 (2022-08-04 15:01:07)

fix: discover product types for new default providers
For new system-wide providers (updated system `providers.yml`), launch product types discovery after having fetched external products types configuration:

If a new provider `foo`, is added by user in `{eodag_install_dir}/resources/providers.yml`, it will not be listed in the remote [`ext_product_types.json`](https://cs-si.github.io/eodag/eodag/resources/ext_product_types.json) which will trigger its product types discovery.

This behavior will also be the same when the providers configuration from eodag github `develop` branch differs from the user one.

See [product types discovery documentation](https://eodag.readthedocs.io/en/latest/notebooks/api_user_guide/2_providers_products_available.html#Product-types-discovery) for more information

github-actions[bot]: ## Unit Test Results
    2 files  ±0      2 suites  ±0   12m 13s [:stopwatch:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "duration of all tests") +13s
270 tests +2  268 [:heavy_check_mark:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "passed tests") +2  2 [:zzz:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "skipped / disabled tests") ±0  0 [:x:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "failed tests") ±0 
540 runs  +4  536 [:heavy_check_mark:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "passed tests") +4  4 [:zzz:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "skipped / disabled tests") ±0  0 [:x:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "failed tests") ±0 

Results for commit 795652e0. ± Comparison against base commit a65a6bca.

[test-results]:data:application/gzip;base64,H4sIAMPh62IC/02Myw6DIBQFf8Ww7oKAcKE/08gVElKVhseq6b8XH6jLmZMzX+L8ZBN5duzRkVR8PmEsccg+LBWB8yrqlLcRaKNXKoirkupSb/85Ertwg5+quE42xhAPE8uyNkXfoCUFl6fZi33jW3Djew/DPPtcgYAWUjBLQWgDg1aUOg6gDNV21FYqjohMGUl+f6NbXBgFAQAA

github-actions[bot]: <strong> </strong>

| File | Coverage |   |
| - | :-: | :-: |
| **All files** | `81%` | :white_check_mark: |
| eodag/api/core.py | `88%` | :white_check_mark: |
| tests/units/test\_core.py | `98%` | :white_check_mark: |

<strong> </strong>

| File | Coverage |   |
| - | :-: | :-: |
| **All files** | `80%` | :white_check_mark: |
| eodag/api/core.py | `88%` | :white_check_mark: |
| tests/units/test\_core.py | `98%` | :white_check_mark: |
| eodag/api/core.py | `88%` | :white_check_mark: |
| tests/units/test\_core.py | `98%` | :white_check_mark: |

_Minimum allowed coverage is `70%`_

<p align="right">Generated by :monkey: cobertura-action against 795652e0759b7a9800f3778b09ed9e683ccc28b6 </p>

Patch:
diff --git a/docs/notebooks/api_user_guide/2_providers_products_available.ipynb b/docs/notebooks/api_user_guide/2_providers_products_available.ipynb
index fb0bb523..2844b78a 100644
--- a/docs/notebooks/api_user_guide/2_providers_products_available.ipynb
+++ b/docs/notebooks/api_user_guide/2_providers_products_available.ipynb
@@ -433,7 +433,7 @@
     "In EODAG, the discovered *EODAG external product types configuration file* can be set to:\n",
     "\n",
     "* a file automatically built from github actions and stored in [eodag/resources/ext_product_types.json](https://raw.githubusercontent.com/CS-SI/eodag/develop/eodag/resources/ext_product_types.json) (default settings)\n",
-    "* a custom remote or local file by setting its path in `EODAG_EXT_PRODUCT_TYPES_CFG_FILE` environment variable.\n",
+    "* a custom remote or local file by setting its path in `EODAG_EXT_PRODUCT_TYPES_CFG_FILE` environment variable (if the file is not readable, only user-modified providers will be fetched).\n",
     "\n",
     "Then, when listing product types using [list_product_types(fetch_providers=True)](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.list_product_types), EODAG will first read the content of the *EODAG external product types configuration file* using [fetch_product_types_list()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.fetch_product_types_list) \n",
     "then update [EODataAccessGateway](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway) instance product types configuration, if needed.\n",
diff --git a/eodag/api/core.py b/eodag/api/core.py
index ffe01d83..c1d50882 100644
--- a/eodag/api/core.py
+++ b/eodag/api/core.py
@@ -594,14 +594,26 @@ class EODataAccessGateway(object):
                     default_provider_search_config, "discover_product_types", {}
                 )
                 # compare confs
-                if default_discovery_conf == user_discovery_conf:
+                if default_discovery_conf == user_discovery_conf and (
+                    not default_discovery_conf.get("fetch_url", None)
+                    or "ext_product_types_conf" not in locals()
+                    or "ext_product_types_conf" in locals()
+                    and (
+                        provider in ext_product_types_conf
+                        or len(ext_product_types_conf.keys()) == 0
+                    )
+                ):
                     continue
+                # providers not skipped here should be user-modified
+                # or not in ext_product_types_conf (if eodag system conf != eodag conf used for ext_product_types_conf)
 
             # discover product types for user configured provider
-            ext_product_types_conf = self.discover_product_types(provider=provider)
+            provider_ext_product_types_conf = self.discover_product_types(
+                provider=provider
+            )
 
             # update eodag product types list with new conf
-            self.update_product_types_list(ext_product_types_conf)
+            self.update_product_types_list(provider_ext_product_types_conf)
 
     def discover_product_types(self, provider=None):
         """Fetch providers for product types


Test patch:
diff --git a/tests/units/test_core.py b/tests/units/test_core.py
index 67d3f9ac..edaac541 100644
--- a/tests/units/test_core.py
+++ b/tests/units/test_core.py
@@ -41,6 +41,7 @@ from tests.context import (
     SearchResult,
     UnsupportedProvider,
     get_geometry_from_various,
+    load_default_config,
     makedirs,
 )
 from tests.utils import mock, write_eodag_conf_with_fake_credentials
@@ -341,6 +342,18 @@ class TestCore(TestCoreBase):
                 "product_types_config": {"foo": {"title": "Foo collection"}},
             }
         }
+        # add an empty ext-conf for other providers to prevent them to be fetched
+        for provider, provider_config in self.dag.providers_config.items():
+            if provider != "astraea_eod" and hasattr(provider_config, "search"):
+                provider_search_config = provider_config.search
+            elif provider != "astraea_eod" and hasattr(provider_config, "api"):
+                provider_search_config = provider_config.api
+            else:
+                continue
+            if hasattr(
+                provider_search_config, "discover_product_types"
+            ) and provider_search_config.discover_product_types.get("fetch_url", None):
+                mock_get_ext_product_types_conf.return_value[provider] = {}
         self.dag.fetch_product_types_list()
         self.assertTrue(self.dag.providers_config["astraea_eod"].product_types_fetched)
         self.assertEqual(
@@ -387,6 +400,85 @@ class TestCore(TestCoreBase):
         # (once per dynamically configured provider)
         self.assertEqual(mock_discover_product_types.call_count, 3)
 
+    @mock.patch("eodag.api.core.get_ext_product_types_conf", autospec=True)
+    @mock.patch(
+        "eodag.api.core.EODataAccessGateway.discover_product_types", autospec=True
+    )
+    def test_fetch_product_types_list_updated_system_conf(
+        self, mock_discover_product_types, mock_get_ext_product_types_conf
+    ):
+        """fetch_product_types_list must launch product types discovery for new system-wide providers"""
+        # add a new system-wide provider not listed in ext-conf
+        new_default_conf = load_default_config()
+        new_default_conf["new_provider"] = new_default_conf["astraea_eod"]
+
+        with mock.patch(
+            "eodag.api.core.load_default_config",
+            return_value=new_default_conf,
+            autospec=True,
+        ):
+            self.dag = EODataAccessGateway()
+
+            mock_get_ext_product_types_conf.return_value = {}
+
+            # disabled product types discovery
+            os.environ["EODAG_EXT_PRODUCT_TYPES_CFG_FILE"] = ""
+            self.dag.fetch_product_types_list()
+            mock_discover_product_types.assert_not_called()
+            os.environ.pop("EODAG_EXT_PRODUCT_TYPES_CFG_FILE")
+
+            # add an empty ext-conf for other providers to prevent them to be fetched
+            for provider, provider_config in self.dag.providers_config.items():
+                if provider != "new_provider" and hasattr(provider_config, "search"):
+                    provider_search_config = provider_config.search
+                elif provider != "new_provider" and hasattr(provider_config, "api"):
+                    provider_search_config = provider_config.api
+                else:
+                    continue
+                if hasattr(
+                    provider_search_config, "discover_product_types"
+                ) and provider_search_config.discover_product_types.get(
+                    "fetch_url", None
+                ):
+                    mock_get_ext_product_types_conf.return_value[provider] = {}
+
+            self.dag.fetch_product_types_list()
+            mock_discover_product_types.assert_called_once_with(
+                self.dag, provider="new_provider"
+            )
+
+    @mock.patch(
+        "eodag.api.core.EODataAccessGateway.discover_product_types", autospec=True
+    )
+    def test_fetch_product_types_list_disabled(self, mock_discover_product_types):
+        """fetch_product_types_list must not launch product types discovery if disabled"""
+
+        # disable product types discovery
+        os.environ["EODAG_EXT_PRODUCT_TYPES_CFG_FILE"] = ""
+
+        # default settings
+        self.dag.fetch_product_types_list()
+        mock_discover_product_types.assert_not_called()
+
+        # only user-defined providers must be fetched
+        self.dag.update_providers_config(
+            """
+            astraea_eod:
+                search:
+                    discover_product_types:
+                        fetch_url: 'http://new-endpoint'
+            foo_provider:
+                search:
+                    type: StacSearch
+                    api_endpoint: https://foo.bar/search
+                products:
+                    GENERIC_PRODUCT_TYPE:
+                        productType: '{productType}'
+            """
+        )
+        self.dag.fetch_product_types_list()
+        self.assertEqual(mock_discover_product_types.call_count, 2)
+
     def assertListProductTypesRightStructure(self, structure):
         """Helper method to verify that the structure given is a good result of
         EODataAccessGateway.list_product_types