Repository: CS-SI/eodag (python)

Issue:
USGS Search by ID lists many products instead of just the matching one
When I try to use `eodag.search` method with setting the `id` parameter, it lists many (millions) of products instead of just the product matching the ID, and the only work around I found is to iterate through all the pages and look for the matching product manually, but it takes a lot of time to load all these results.

**Code To Reproduce**
```py
from eodag import EODataAccessGateway
from eodag.utils.logging import setup_logging
setup_logging(verbose=3)
dag = EODataAccessGateway()
dag.search(id="LC09_L2SP_016035_20240605_20240611_02_T1", provider="usgs", productType="LANDSAT_C2L2")
```

**Output**
```
2024-07-08 22:46:20,166 eodag.config                     [DEBUG   ] (tid=140046769037376) Loading configuration from /home/hamed/Desktop/playground/env/lib/python3.11/site-packages/eodag/resources/providers.yml
2024-07-08 22:46:21,432 eodag.config                     [INFO    ] (tid=140046769037376) Loading user configuration from: /home/hamed/.config/eodag/eodag.yml
2024-07-08 22:46:21,452 eodag.core                       [INFO    ] (tid=140046769037376) aws_eos: provider needing auth for search has been pruned because no crendentials could be found
2024-07-08 22:46:21,452 eodag.core                       [INFO    ] (tid=140046769037376) meteoblue: provider needing auth for search has been pruned because no crendentials could be found
2024-07-08 22:46:21,452 eodag.core                       [INFO    ] (tid=140046769037376) hydroweb_next: provider needing auth for search has been pruned because no crendentials could be found
2024-07-08 22:46:21,452 eodag.core                       [INFO    ] (tid=140046769037376) wekeo: provider needing auth for search has been pruned because no crendentials could be found
2024-07-08 22:46:21,452 eodag.core                       [INFO    ] (tid=140046769037376) creodias_s3: provider needing auth for search has been pruned because no crendentials could be found
2024-07-08 22:46:21,460 eodag.core                       [DEBUG   ] (tid=140046769037376) Opening product types index in /home/hamed/.config/eodag/.index
2024-07-08 22:46:21,473 eodag.core                       [INFO    ] (tid=140046769037376) Locations configuration loaded from /home/hamed/.config/eodag/locations.yml
2024-07-08 22:46:21,474 eodag.core                       [INFO    ] (tid=140046769037376) Searching product type 'LANDSAT_C2L2' on provider: usgs
2024-07-08 22:46:21,474 eodag.core                       [INFO    ] (tid=140046769037376) Searching product with id 'LC09_L2SP_016035_20240605_20240611_02_T1' on provider: usgs
2024-07-08 22:46:21,474 eodag.core                       [DEBUG   ] (tid=140046769037376) Using plugin class for search: UsgsApi
2024-07-08 22:46:22,680 eodag.apis.usgs                  [INFO    ] (tid=140046769037376) Sending search request for landsat_ot_c2_l2 with {'start_date': None, 'end_date': None, 'll': None, 'ur': None, 'max_results': 2, 'starting_number': 1}
2024-07-08 22:46:24,927 eodag.apis.usgs                  [DEBUG   ] (tid=140046769037376) Adapting 2 plugin results to eodag product representation
2024-07-08 22:46:27,095 eodag.core                       [INFO    ] (tid=140046769037376) Found 3127730 result(s) on provider 'usgs'
2024-07-08 22:46:27,095 eodag.core                       [INFO    ] (tid=140046769037376) Several products found for this id ([EOProduct(id=LC09_L2SP_016001_20240707_20240708_02_T1, provider=usgs), EOProduct(id=LC09_L2SP_016002_20240707_20240708_02_T1, provider=usgs)]). You may try searching using more selective criteria.
```

**Environment:**
 - Python version: `3.11.2`
 - EODAG version: `2.12.1`

**Additional context**
Here is the actual product that corresponds to the "ID" in the search:
https://earthexplorer.usgs.gov/scene/metadata/full/5e83d14f2fc39685/LC90160352024157LGN00/


Pull request:
fix(providers): usgs search by id
Fixes #1252

Implements lacking search-by-id mechanism on `UsgsApi` plugin for `usgs` provider

Patch:
diff --git a/eodag/plugins/apis/usgs.py b/eodag/plugins/apis/usgs.py
index 2e552f0b..d4ecf61f 100644
--- a/eodag/plugins/apis/usgs.py
+++ b/eodag/plugins/apis/usgs.py
@@ -172,11 +172,39 @@ class UsgsApi(Api):
                 max_results=items_per_page,
                 starting_number=(1 + (page - 1) * items_per_page),
             )
-            logger.info(
-                f"Sending search request for {usgs_dataset} with {api_search_kwargs}"
-            )
 
-            results = api.scene_search(usgs_dataset, **api_search_kwargs)
+            # search by id
+            if searched_id := kwargs.get("id"):
+                dataset_filters = api.dataset_filters(usgs_dataset)
+                # ip pattern set as parameter queryable (first element of param conf list)
+                id_pattern = self.config.metadata_mapping["id"][0]
+                # loop on matching dataset_filters until one returns expected results
+                for dataset_filter in dataset_filters["data"]:
+                    if id_pattern in dataset_filter["searchSql"]:
+                        logger.debug(
+                            f"Try using {dataset_filter['searchSql']} dataset filter to search by id on {usgs_dataset}"
+                        )
+                        full_api_search_kwargs = {
+                            "where": {
+                                "filter_id": dataset_filter["id"],
+                                "value": searched_id,
+                            },
+                            **api_search_kwargs,
+                        }
+                        logger.info(
+                            f"Sending search request for {usgs_dataset} with {full_api_search_kwargs}"
+                        )
+                        results = api.scene_search(
+                            usgs_dataset, **full_api_search_kwargs
+                        )
+                        if len(results["data"]["results"]) == 1:
+                            # search by id using this dataset_filter succeeded
+                            break
+            else:
+                logger.info(
+                    f"Sending search request for {usgs_dataset} with {api_search_kwargs}"
+                )
+                results = api.scene_search(usgs_dataset, **api_search_kwargs)
 
             # update results with storage info from download_options()
             results_by_entity_id = {
diff --git a/eodag/resources/providers.yml b/eodag/resources/providers.yml
index 51fc63f2..b2013f83 100644
--- a/eodag/resources/providers.yml
+++ b/eodag/resources/providers.yml
@@ -31,7 +31,9 @@
       max_items_per_page: 5000
       total_items_nb_key_path: '$.totalHits'
     metadata_mapping:
-      id: '$.displayId'
+      id:
+        - '_ID'
+        - '$.displayId'
       geometry: '$.spatialBounds'
       productType: '$.productType'
       title: '$.displayId'


Test patch:
diff --git a/tests/units/test_apis_plugins.py b/tests/units/test_apis_plugins.py
index 30cfbb41..7f3d184a 100644
--- a/tests/units/test_apis_plugins.py
+++ b/tests/units/test_apis_plugins.py
@@ -392,6 +392,65 @@ class TestApisPluginEcmwfApi(BaseApisPluginTest):
 
 
 class TestApisPluginUsgsApi(BaseApisPluginTest):
+    SCENE_SEARCH_RETURN = {
+        "data": {
+            "results": [
+                {
+                    "browse": [
+                        {
+                            "browsePath": "https://path/to/quicklook.jpg",
+                            "thumbnailPath": "https://path/to/thumbnail.jpg",
+                        },
+                        {},
+                        {},
+                    ],
+                    "cloudCover": "77.46",
+                    "entityId": "LC81780382020041LGN00",
+                    "displayId": "LC08_L1GT_178038_20200210_20200224_01_T2",
+                    "spatialBounds": {
+                        "type": "Polygon",
+                        "coordinates": [
+                            [
+                                [28.03905, 30.68073],
+                                [28.03905, 32.79057],
+                                [30.46294, 32.79057],
+                                [30.46294, 30.68073],
+                                [28.03905, 30.68073],
+                            ]
+                        ],
+                    },
+                    "temporalCoverage": {
+                        "endDate": "2020-02-10 00:00:00",
+                        "startDate": "2020-02-10 00:00:00",
+                    },
+                    "publishDate": "2020-02-10 08:24:46",
+                },
+            ],
+            "recordsReturned": 5,
+            "totalHits": 139,
+            "startingNumber": 6,
+            "nextRecord": 11,
+        }
+    }
+
+    DOWNLOAD_OPTION_RETURN = {
+        "data": [
+            {
+                "id": "5e83d0b8e7f6734c",
+                "entityId": "LC81780382020041LGN00",
+                "available": True,
+                "filesize": 9186067,
+                "productName": "LandsatLook Natural Color Image",
+                "downloadSystem": "dds",
+            },
+            {
+                "entityId": "LC81780382020041LGN00",
+                "available": False,
+                "downloadSystem": "wms",
+            },
+        ]
+    }
+
     def setUp(self):
         self.provider = "usgs"
         self.api_plugin = self.get_search_plugin(provider=self.provider)
@@ -450,67 +509,12 @@ class TestApisPluginUsgsApi(BaseApisPluginTest):
     @mock.patch(
         "usgs.api.scene_search",
         autospec=True,
-        return_value={
-            "data": {
-                "results": [
-                    {
-                        "browse": [
-                            {
-                                "browsePath": "https://path/to/quicklook.jpg",
-                                "thumbnailPath": "https://path/to/thumbnail.jpg",
-                            },
-                            {},
-                            {},
-                        ],
-                        "cloudCover": "77.46",
-                        "entityId": "LC81780382020041LGN00",
-                        "displayId": "LC08_L1GT_178038_20200210_20200224_01_T2",
-                        "spatialBounds": {
-                            "type": "Polygon",
-                            "coordinates": [
-                                [
-                                    [28.03905, 30.68073],
-                                    [28.03905, 32.79057],
-                                    [30.46294, 32.79057],
-                                    [30.46294, 30.68073],
-                                    [28.03905, 30.68073],
-                                ]
-                            ],
-                        },
-                        "temporalCoverage": {
-                            "endDate": "2020-02-10 00:00:00",
-                            "startDate": "2020-02-10 00:00:00",
-                        },
-                        "publishDate": "2020-02-10 08:24:46",
-                    },
-                ],
-                "recordsReturned": 5,
-                "totalHits": 139,
-                "startingNumber": 6,
-                "nextRecord": 11,
-            }
-        },
+        return_value=SCENE_SEARCH_RETURN,
     )
     @mock.patch(
         "usgs.api.download_options",
         autospec=True,
-        return_value={
-            "data": [
-                {
-                    "id": "5e83d0b8e7f6734c",
-                    "entityId": "LC81780382020041LGN00",
-                    "available": True,
-                    "filesize": 9186067,
-                    "productName": "LandsatLook Natural Color Image",
-                    "downloadSystem": "dds",
-                },
-                {
-                    "entityId": "LC81780382020041LGN00",
-                    "available": False,
-                    "downloadSystem": "wms",
-                },
-            ]
-        },
+        return_value=DOWNLOAD_OPTION_RETURN,
     )
     def test_plugins_apis_usgs_query(
         self,
@@ -566,6 +570,61 @@ class TestApisPluginUsgsApi(BaseApisPluginTest):
             total_count, mock_api_scene_search.return_value["data"]["totalHits"]
         )
 
+    @mock.patch("usgs.api.login", autospec=True)
+    @mock.patch("usgs.api.logout", autospec=True)
+    @mock.patch(
+        "usgs.api.scene_search",
+        autospec=True,
+        return_value=SCENE_SEARCH_RETURN,
+    )
+    @mock.patch(
+        "usgs.api.download_options",
+        autospec=True,
+        return_value=DOWNLOAD_OPTION_RETURN,
+    )
+    @mock.patch(
+        "usgs.api.dataset_filters",
+        autospec=True,
+        return_value={
+            "data": [
+                {"id": "foo_id", "searchSql": "DONT_USE_THIS !"},
+                {"id": "bar_id", "searchSql": "USE_THIS_ID !"},
+            ]
+        },
+    )
+    def test_plugins_apis_usgs_query_by_id(
+        self,
+        mock_dataset_filters,
+        mock_api_download_options,
+        mock_api_scene_search,
+        mock_api_logout,
+        mock_api_login,
+    ):
+        """UsgsApi.query by id must search using usgs api"""
+
+        search_kwargs = {
+            "productType": "LANDSAT_C2L1",
+            "id": "SOME_PRODUCT_ID",
+            "prep": PreparedSearch(
+                items_per_page=500,
+                page=1,
+            ),
+        }
+        search_results, total_count = self.api_plugin.query(**search_kwargs)
+        mock_api_scene_search.assert_called_once_with(
+            "landsat_ot_c2_l1",
+            where={"filter_id": "bar_id", "value": "SOME_PRODUCT_ID"},
+            start_date=None,
+            end_date=None,
+            ll=None,
+            ur=None,
+            max_results=500,
+            starting_number=1,
+        )
+        self.assertEqual(search_results[0].provider, "usgs")
+        self.assertEqual(search_results[0].product_type, "LANDSAT_C2L1")
+        self.assertEqual(len(search_results), 1)
+
     @mock.patch("usgs.api.login", autospec=True)
     @mock.patch("usgs.api.logout", autospec=True)
     @mock.patch("usgs.api.download_request", autospec=True)