Repository: CS-SI/eodag (python)

Issue:
onda search optimize with expand=Metadata
Avoid per-product metadata query using https://catalogue.onda-dias.eu/dias-catalogue/Products?$expand=Metadata link documented in 
https://www.onda-dias.eu/cms/knowledge-base/odata-querying-products-showing-all-metadata-together-with-their-properties/.

Use a new `ODataV4Search` plugin configuration option, `per_product_metadata_query` (bool) to specify that per-product metadata query is no more needed.

See also https://github.com/CS-SI/eodag/issues/606#issuecomment-1399962814

Pull request description:
feat: optimize onda search and update discover_metadata mechanism
Fixes #613 

- optimizes onda search using `$expand=Metadata` instead of per product metadata query
- updates discover_metadata mechanism to be compatible with this updated metadata formatting

Patch:
diff --git a/docs/_static/params_mapping_opensearch.csv b/docs/_static/params_mapping_opensearch.csv
index 3e98c49f..053fe5dc 100644
--- a/docs/_static/params_mapping_opensearch.csv
+++ b/docs/_static/params_mapping_opensearch.csv
@@ -8,7 +8,7 @@ antennaLookDirection,,,,,,,,,metadata only,,,,metadata only,
 archivingCenter,,,,,,,,,metadata only,,,,metadata only,
 availabilityTime,metadata only,,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,,,,metadata only,:green:`queryable metadata`
 :abbr:`classification ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] Name of the handling restrictions on the resource or metadata (String ))`,,,,,,,,,metadata only,,,,,
-cloudCover,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
+cloudCover,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
 completionTimeFromAscendingNode,:green:`queryable metadata`,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
 :abbr:`compositeType ([OpenSearch Parameters for Collection Search] Type of composite product expressed as time period that the composite product covers (e.g. P10D for a 10 day composite) (String))`,,,,,,,,,metadata only,,,,,
 creationDate,metadata only,,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,metadata only,metadata only,metadata only,metadata only,:green:`queryable metadata`
@@ -22,42 +22,42 @@ illuminationAzimuthAngle,metadata only,,,,:green:`queryable metadata`,:green:`qu
 illuminationElevationAngle,metadata only,,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,,,,metadata only,:green:`queryable metadata`
 illuminationZenithAngle,,,,,,,,,metadata only,,,,metadata only,
 incidenceAngleVariation,,,,,,,,,metadata only,,,,metadata only,
-":abbr:`instrument ([OpenSearch Parameters for Collection Search] A string identifying the instrument (e.g. MERIS, AATSR, ASAR, HRVIR. SAR). (String))`",metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
+":abbr:`instrument ([OpenSearch Parameters for Collection Search] A string identifying the instrument (e.g. MERIS, AATSR, ASAR, HRVIR. SAR). (String))`",metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
 :abbr:`keyword ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] Commonly used word(s) or formalised word(s) or phrase(s) used to describe the subject. (String))`,,,,metadata only,,,,,metadata only,,metadata only,metadata only,metadata only,
 :abbr:`language ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] Language of the intellectual content of the metadata record (String ))`,,,,,,,,,metadata only,,,,,
-:abbr:`lineage ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] General explanation of the data producer’s knowledge about the lineage of a dataset. (String))`,,,,,,,,,metadata only,,,,,
+:abbr:`lineage ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] General explanation of the data producer’s knowledge about the lineage of a dataset. (String))`,,,,,,,,,metadata only,:green:`queryable metadata`,,,,
 lowestLocation,,,,,,,,,metadata only,,,,,
 maximumIncidenceAngle,,,,,,,,,metadata only,,,,metadata only,
 minimumIncidenceAngle,,,,,,,,,metadata only,,,,metadata only,
 modificationDate,metadata only,,,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,,metadata only,metadata only,metadata only,:green:`queryable metadata`
-orbitDirection,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`
-orbitNumber,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,:green:`queryable metadata`
+orbitDirection,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`
+orbitNumber,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,:green:`queryable metadata`
 ":abbr:`orbitType ([OpenSearch Parameters for Collection Search] A string identifying the platform orbit type (e.g. LEO, GEO) (String))`",,,,,,,,,metadata only,,,,,
 :abbr:`organisationName ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] A string identifying the name of the organization responsible for the resource (String))`,,,,:green:`queryable metadata`,,,,,metadata only,,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,
 :abbr:`organisationRole ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] The function performed by the responsible party (String ))`,,,,,,,,,metadata only,,,,,
 :abbr:`otherConstraint ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] Other restrictions and legal prerequisites for accessing and using the resource or metadata. (String))`,,,,,,,,,metadata only,,,,,
 parentIdentifier,,,,:green:`queryable metadata`,,,,,metadata only,,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,
 :abbr:`platform ([OpenSearch Parameters for Collection Search] A string with the platform short name (e.g. Sentinel-1) (String))`,metadata only,,,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,:green:`queryable metadata`,metadata only,metadata only,:green:`queryable metadata`,:green:`queryable metadata`
-:abbr:`platformSerialIdentifier ([OpenSearch Parameters for Collection Search] A string with the Platform serial identifier (String))`,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,:green:`queryable metadata`,,metadata only,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
+:abbr:`platformSerialIdentifier ([OpenSearch Parameters for Collection Search] A string with the Platform serial identifier (String))`,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,:green:`queryable metadata`,,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
 processingCenter,,,,,,,,,metadata only,,metadata only,metadata only,,
 processingDate,,,,,,,,,metadata only,metadata only,,,metadata only,
-:abbr:`processingLevel ([OpenSearch Parameters for Collection Search] A string identifying the processing level applied to the entry (String))`,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
+:abbr:`processingLevel ([OpenSearch Parameters for Collection Search] A string identifying the processing level applied to the entry (String))`,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
 processingMode,,,,,,,,,metadata only,,,,metadata only,
 processorName,,,,,,,,,metadata only,,metadata only,metadata only,,
 productQualityDegradationTag,,,,,,,,,metadata only,,,,,
-productQualityStatus,,,,,,,,,metadata only,metadata only,metadata only,metadata only,,
+productQualityStatus,,,,,,,,,metadata only,:green:`queryable metadata`,metadata only,metadata only,,
 ":abbr:`productType ([OpenSearch Parameters for Collection Search] A string identifying the entry type (e.g. ER02_SAR_IM__0P, MER_RR__1P, SM_SLC__1S, GES_DISC_AIRH3STD_V005) (String ))`",:green:`queryable metadata`,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`
-productVersion,metadata only,,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,metadata only,metadata only,metadata only,metadata only,:green:`queryable metadata`
+productVersion,metadata only,,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,:green:`queryable metadata`,metadata only,metadata only,metadata only,:green:`queryable metadata`
 :abbr:`publicationDate ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] The date when the resource was issued (Date time))`,metadata only,,,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,,metadata only,metadata only,metadata only,:green:`queryable metadata`
 resolution,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,:green:`queryable metadata`
-sensorMode,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,:green:`queryable metadata`
-":abbr:`sensorType ([OpenSearch Parameters for Collection Search] A string identifying the sensor type. Suggested values are: OPTICAL, RADAR, ALTIMETRIC, ATMOSPHERIC, LIMB (String))`",,,,,,,,,metadata only,,,,,
+sensorMode,metadata only,,,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,:green:`queryable metadata`
+":abbr:`sensorType ([OpenSearch Parameters for Collection Search] A string identifying the sensor type. Suggested values are: OPTICAL, RADAR, ALTIMETRIC, ATMOSPHERIC, LIMB (String))`",,,,,,,,,metadata only,:green:`queryable metadata`,,,,
 snowCover,,,,:green:`queryable metadata`,,,,,metadata only,,:green:`queryable metadata`,:green:`queryable metadata`,metadata only,
 ":abbr:`spectralRange ([OpenSearch Parameters for Collection Search] A string identifying the sensor spectral range (e.g. INFRARED, NEAR-INFRARED, UV, VISIBLE) (String))`",,,,,,,,,metadata only,,,,,
 startTimeFromAscendingNode,metadata only,,,:green:`queryable metadata`,metadata only,metadata only,metadata only,,:green:`queryable metadata`,metadata only,:green:`queryable metadata`,:green:`queryable metadata`,:green:`queryable metadata`,metadata only
 swathIdentifier,,,,,,,,,metadata only,,:green:`queryable metadata`,:green:`queryable metadata`,,
 :abbr:`title ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] A name given to the resource (String ))`,metadata only,,,metadata only,metadata only,metadata only,metadata only,,metadata only,metadata only,metadata only,metadata only,metadata only,metadata only
-:abbr:`topicCategory ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] Main theme(s) of the dataset (String ))`,,,,,,,,,metadata only,,metadata only,metadata only,,
+:abbr:`topicCategory ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] Main theme(s) of the dataset (String ))`,,,,,,,,,metadata only,:green:`queryable metadata`,metadata only,metadata only,,
 track,,,,,,,,,metadata only,,,,,
 :abbr:`useLimitation ([Additional INSPIRE obligated OpenSearch Parameters for Collection Search] A string identifying informing if the resource has usage limitations (String))`,,,,,,,,,metadata only,,,,,
 wavelengths,,,,,,,,,metadata only,,,,,
diff --git a/eodag/api/product/metadata_mapping.py b/eodag/api/product/metadata_mapping.py
index 570213d6..cdb0c2ce 100644
--- a/eodag/api/product/metadata_mapping.py
+++ b/eodag/api/product/metadata_mapping.py
@@ -25,6 +25,7 @@ from string import Formatter
 import geojson
 from dateutil.parser import isoparse
 from dateutil.tz import UTC, tzutc
+from jsonpath_ng.jsonpath import Child
 from lxml import etree
 from lxml.etree import XPathEvalError
 from shapely import wkt
@@ -407,7 +408,7 @@ def format_metadata(search_param, *args, **kwargs):
     return MetadataFormatter().vformat(search_param, args, kwargs)
 
 
-def properties_from_json(json, mapping, discovery_pattern=None, discovery_path=None):
+def properties_from_json(json, mapping, discovery_config=None):
     """Extract properties from a provider json result.
 
     :param json: The representation of a provider result as a json object
@@ -442,7 +443,7 @@ def properties_from_json(json, mapping, discovery_pattern=None, discovery_path=N
             match = path_or_text.find(json)
             if len(match) == 1:
                 extracted_value = match[0].value
-                used_jsonpaths.append(match[0].path)
+                used_jsonpaths.append(match[0].full_path)
             else:
                 extracted_value = NOT_AVAILABLE
             if extracted_value is None:
@@ -483,16 +484,43 @@ def properties_from_json(json, mapping, discovery_pattern=None, discovery_path=N
         properties[metadata] = template.format(**properties)
 
     # adds missing discovered properties
+    if not discovery_config:
+        discovery_config = {}
+    discovery_pattern = discovery_config.get("metadata_pattern", None)
+    discovery_path = discovery_config.get("metadata_path", None)
     if discovery_pattern and discovery_path:
         discovered_properties = cached_parse(discovery_path).find(json)
         for found_jsonpath in discovered_properties:
-            found_key = found_jsonpath.path.fields[-1]
+            if "metadata_path_id" in discovery_config.keys():
+                found_key_paths = cached_parse(
+                    discovery_config["metadata_path_id"]
+                ).find(found_jsonpath.value)
+                if not found_key_paths:
+                    continue
+                found_key = found_key_paths[0].value
+                used_jsonpath = Child(
+                    found_jsonpath.full_path,
+                    cached_parse(discovery_config["metadata_path_value"]),
+                )
+            else:
+                # default key got from metadata_path
+                found_key = found_jsonpath.path.fields[-1]
+                used_jsonpath = found_jsonpath.full_path
             if (
                 re.compile(discovery_pattern).match(found_key)
                 and found_key not in properties.keys()
-                and found_jsonpath.path not in used_jsonpaths
+                and used_jsonpath not in used_jsonpaths
             ):
-                properties[found_key] = found_jsonpath.value
+                if "metadata_path_value" in discovery_config.keys():
+                    found_value_path = cached_parse(
+                        discovery_config["metadata_path_value"]
+                    ).find(found_jsonpath.value)
+                    properties[found_key] = (
+                        found_value_path[0].value if found_value_path else NOT_AVAILABLE
+                    )
+                else:
+                    # default value got from metadata_path
+                    properties[found_key] = found_jsonpath.value
 
     return properties
 
diff --git a/eodag/plugins/search/build_search_result.py b/eodag/plugins/search/build_search_result.py
index 84b5b742..d978f99f 100644
--- a/eodag/plugins/search/build_search_result.py
+++ b/eodag/plugins/search/build_search_result.py
@@ -121,12 +121,7 @@ class BuildPostSearchResult(PostJsonSearch):
         parsed_properties = properties_from_json(
             result,
             self.config.metadata_mapping,
-            discovery_pattern=getattr(self.config, "discover_metadata", {}).get(
-                "metadata_pattern", None
-            ),
-            discovery_path=getattr(self.config, "discover_metadata", {}).get(
-                "metadata_path", "null"
-            ),
+            discovery_config=getattr(self.config, "discover_metadata", {}),
         )
 
         if not product_type:
diff --git a/eodag/plugins/search/qssearch.py b/eodag/plugins/search/qssearch.py
index ec911bb7..9a48a2ac 100644
--- a/eodag/plugins/search/qssearch.py
+++ b/eodag/plugins/search/qssearch.py
@@ -732,12 +732,7 @@ class QueryStringSearch(Search):
                 QueryStringSearch.extract_properties[self.config.result_type](
                     result,
                     self.config.metadata_mapping,
-                    discovery_pattern=getattr(self.config, "discover_metadata", {}).get(
-                        "metadata_pattern", None
-                    ),
-                    discovery_path=getattr(self.config, "discover_metadata", {}).get(
-                        "metadata_path", "null"
-                    ),
+                    discovery_config=getattr(self.config, "discover_metadata", {}),
                 ),
                 **kwargs,
             )
@@ -956,29 +951,31 @@ class ODataV4Search(QueryStringSearch):
     all products metadata"""
 
     def do_search(self, *args, **kwargs):
-        """Do a two step search, as the metadata are not given into the search result"""
-        # TODO: This plugin is still very specific to the ONDA provider.
-        #       Be careful to generalize it if needed when the chance to do so arrives
-        final_result = []
-        # Query the products entity set for basic metadata about the product
-        for entity in super(ODataV4Search, self).do_search(*args, **kwargs):
-            metadata_url = self.get_metadata_search_url(entity)
-            try:
-                logger.debug("Sending metadata request: %s", metadata_url)
-                response = requests.get(metadata_url, timeout=HTTP_REQ_TIMEOUT)
-                response.raise_for_status()
-            except requests.RequestException:
-                logger.exception(
-                    "Skipping error while searching for %s %s instance:",
-                    self.provider,
-                    self.__class__.__name__,
-                )
-            else:
-                entity.update(
-                    {item["id"]: item["value"] for item in response.json()["value"]}
-                )
-                final_result.append(entity)
-        return final_result
+        """A two step search can be performed if the metadata are not given into the search result"""
+
+        if getattr(self.config, "per_product_metadata_query", False):
+            final_result = []
+            # Query the products entity set for basic metadata about the product
+            for entity in super(ODataV4Search, self).do_search(*args, **kwargs):
+                metadata_url = self.get_metadata_search_url(entity)
+                try:
+                    logger.debug("Sending metadata request: %s", metadata_url)
+                    response = requests.get(metadata_url, timeout=HTTP_REQ_TIMEOUT)
+                    response.raise_for_status()
+                except requests.RequestException:
+                    logger.exception(
+                        "Skipping error while searching for %s %s instance:",
+                        self.provider,
+                        self.__class__.__name__,
+                    )
+                else:
+                    entity.update(
+                        {item["id"]: item["value"] for item in response.json()["value"]}
+                    )
+                    final_result.append(entity)
+            return final_result
+        else:
+            return super(ODataV4Search, self).do_search(*args, **kwargs)
 
     def get_metadata_search_url(self, entity):
         """Build the metadata link for the given entity"""
diff --git a/eodag/resources/providers.yml b/eodag/resources/providers.yml
index 6ac69347..5f0d4436 100644
--- a/eodag/resources/providers.yml
+++ b/eodag/resources/providers.yml
@@ -1315,7 +1315,7 @@
       - ':'
     pagination:
       count_endpoint: 'https://catalogue.onda-dias.eu/dias-catalogue/Products/$count'
-      next_page_url_tpl: '{url}?{search}&$top={items_per_page}&$skip={skip}'
+      next_page_url_tpl: '{url}?{search}&$top={items_per_page}&$skip={skip}&$expand=Metadata'
       # 2021/03/19: 2000 is the max, if greater 200 response but contains an error message
       max_items_per_page: 2_000
     results_entry: 'value'
@@ -1334,6 +1334,17 @@
             - 'beginPosition:[{startTimeFromAscendingNode#to_iso_utc_datetime} TO *]'
             - 'endPosition:[* TO {completionTimeFromAscendingNode#to_iso_utc_datetime}]'
             - '{id#remove_extension}'
+            - 'platformSerialIdentifier:{platformSerialIdentifier}'
+            - 'instrumentShortName:{instrument}'
+            - 'processingLevel:{processingLevel}'
+            - 'sensorType:{sensorType}'
+            - 'topicCategory:{topicCategory}'
+            - 'lineage:{lineage}'
+            - 'orbitNumber:{orbitNumber}'
+            - 'orbitDirection:{orbitDirection}'
+            - 'processingBaseline:{productVersion}'
+            - 'generalQualityFlag:{productQualityStatus}'
+            - 'sensorOperationalMode:{sensorMode}'
     discover_metadata:
       auto_discovery: true
       metadata_pattern: '^[a-zA-Z0-9]+$'
@@ -1343,42 +1354,74 @@
             operations:
               AND:
                 - '{metadata}:{{{metadata}}}'
-      metadata_path: '$.*'
+      metadata_path: '$.Metadata[*]'
+      metadata_path_id: 'id'
+      metadata_path_value: 'value'
+    per_product_metadata_query: false
     metadata_mapping:
       # Opensearch resource identifier within the search engine context (in our case
       # within the context of the data provider)
+      # Queryable parameters are set with null as 1st configuration list value to mark them as queryable,
+      #   but `free_text_search_operations.$search.operations.AND` entries are then used instead.
       uid: '$.id'
       # OpenSearch Parameters for Collection Search (Table 3)
       productType:
-        - 'productType'
-        - '$.productType'
+        - null
+        - '$.Metadata[?(@.id="productType")].value'
       platform:
-        - platform
-        - '$.platformName'
-      platformSerialIdentifier: '$.platformSerialIdentifier'
-      instrument: '$.instrumentName'
-      processingLevel: '$.processingLevel'
+        - null
+        - '$.Metadata[?(@.id="platformName")].value'
+      platformSerialIdentifier:
+        - null
+        - '$.Metadata[?(@.id="platformSerialIdentifier")].value'
+      instrument:
+        - null
+        - '$.Metadata[?(@.id="instrumentShortName")].value'
+      processingLevel:
+        - null
+        - '$.Metadata[?(@.id="processingLevel")].value'
+      sensorType:
+        - null
+        - '$.Metadata[?(@.id="sensorType")].value'
 
       # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
-      title: '$.filename'
+      title: '$.Metadata[?(@.id="filename")].value'
+      topicCategory:
+        - null
+        - '$.Metadata[?(@.id="topicCategory")].value'
+      lineage:
+        - null
+        - '$.Metadata[?(@.id="lineage")].value'
 
       # OpenSearch Parameters for Product Search (Table 5)
-      orbitNumber: '$.orbitNumber'
-      orbitDirection: '$.orbitDirection'
-      cloudCover: '$.cloudCoverPercentage'
-      productVersion: '$.processingBaseline'
-      productQualityStatus: '$.generalQualityFlag'
-      creationDate: '$.creationDate'
-      processingDate: '$.processingDate'
-      sensorMode: '$.sensorOperationalMode'
+      orbitNumber:
+        - null
+        - '$.Metadata[?(@.id="orbitNumber")].value'
+      orbitDirection:
+        - null
+        - '$.Metadata[?(@.id="orbitDirection")].value'
+      cloudCover:
+        - null
+        - '$.Metadata[?(@.id="cloudCoverPercentage")].value'
+      productVersion:
+        - null
+        - '$.Metadata[?(@.id="processingBaseline")].value'
+      productQualityStatus:
+        - null
+        - '$.Metadata[?(@.id="generalQualityFlag")].value'
+      creationDate: '$.Metadata[?(@.id="creationDate")].value'
+      processingDate: '$.Metadata[?(@.id="processingDate")].value'
+      sensorMode:
+        - null
+        - '$.Metadata[?(@.id="sensorOperationalMode")].value'
 
       # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
       startTimeFromAscendingNode: '$.beginPosition'
       completionTimeFromAscendingNode: '$.endPosition'
-      polarizationChannels: '{$.polarisationChannels#replace_str(","," ")}'
+      polarizationChannels: '{$.Metadata[?(@.id="polarisationChannels")].value#replace_str(","," ")}'
 
       # Custom parameters (not defined in the base document referenced above)
-      id: '{$.filename#remove_extension}'
+      id: '{$.Metadata[?(@.id="filename")].value#remove_extension}'
       # The geographic extent of the product
       geometry: '$.footprint'
       # The url of the quicklook


Test patch:
diff --git a/tests/context.py b/tests/context.py
index dec2c197..2ca8dd32 100644
--- a/tests/context.py
+++ b/tests/context.py
@@ -33,6 +33,8 @@ from eodag.api.product.metadata_mapping import (
     format_metadata,
     OFFLINE_STATUS,
     ONLINE_STATUS,
+    properties_from_json,
+    NOT_AVAILABLE,
 )
 from eodag.api.search_result import SearchResult
 from eodag.cli import download, eodag, list_pt, search_crunch
diff --git a/tests/units/test_metadata_mapping.py b/tests/units/test_metadata_mapping.py
index ba8e15e5..4089601a 100644
--- a/tests/units/test_metadata_mapping.py
+++ b/tests/units/test_metadata_mapping.py
@@ -17,7 +17,14 @@
 # limitations under the License.
 import unittest
 
-from tests.context import format_metadata, get_geometry_from_various
+from jsonpath_ng.ext import parse
+
+from tests.context import (
+    NOT_AVAILABLE,
+    format_metadata,
+    get_geometry_from_various,
+    properties_from_json,
+)
 
 
 class TestMetadataFormatter(unittest.TestCase):
@@ -195,3 +202,67 @@ class TestMetadataFormatter(unittest.TestCase):
             format_metadata(to_format, **{"some_extension:a_parameter": "value"}),
             "value",
         )
+
+    def test_properties_from_json_discovery_config(self):
+        """properties_from_json must extract and discover metadata"""
+        json = {
+            "foo": "foo-val",
+            "bar": "bar-val",
+            "baz": {"baaz": "baz-val"},
+            "qux": [
+                {"somekey": "a", "someval": "a-val"},
+                {"somekey": "b", "someval": "b-val", "some": "thing"},
+                {"somekey": "c"},
+                {"someval": "d-val"},
+            ],
+            "ignored": "ignored-val",
+        }
+        mapping = {
+            "fooProperty": (None, parse("$.foo")),
+            "missingProperty": (None, parse("$.missing")),
+        }
+        # basic discovery
+        discovery_config = {
+            "auto_discovery": True,
+            "metadata_pattern": r"^(?!ignored)[a-zA-Z0-9_]+$",
+            "metadata_path": "$.*",
+        }
+        properties = properties_from_json(
+            json=json, mapping=mapping, discovery_config=discovery_config
+        )
+        self.assertDictEqual(
+            properties,
+            {
+                "fooProperty": "foo-val",
+                "bar": "bar-val",
+                "baz": {"baaz": "baz-val"},
+                "missingProperty": NOT_AVAILABLE,
+                "qux": [
+                    {"somekey": "a", "someval": "a-val"},
+                    {"somekey": "b", "someval": "b-val", "some": "thing"},
+                    {"somekey": "c"},
+                    {"someval": "d-val"},
+                ],
+            },
+        )
+        # advanced discovery
+        discovery_config = {
+            "auto_discovery": True,
+            "metadata_pattern": r"^(?!ignored)[a-zA-Z0-9_]+$",
+            "metadata_path": "$.qux[*]",
+            "metadata_path_id": "somekey",
+            "metadata_path_value": "someval",
+        }
+        properties = properties_from_json(
+            json=json, mapping=mapping, discovery_config=discovery_config
+        )
+        self.assertDictEqual(
+            properties,
+            {
+                "fooProperty": "foo-val",
+                "missingProperty": NOT_AVAILABLE,
+                "a": "a-val",
+                "b": "b-val",
+                "c": NOT_AVAILABLE,
+            },
+        )
diff --git a/tests/units/test_search_plugins.py b/tests/units/test_search_plugins.py
index 16ec3d41..f02accbf 100644
--- a/tests/units/test_search_plugins.py
+++ b/tests/units/test_search_plugins.py
@@ -385,7 +385,13 @@ class TestSearchPluginODataV4Search(BaseSearchPluginTest):
         # Some expected results
         with open(self.provider_resp_dir / "onda_count.json") as f:
             self.onda_resp_count = json.load(f)
-        self.onda_url_count = 'https://catalogue.onda-dias.eu/dias-catalogue/Products/$count?productType=S2MSI1C&$search="footprint:"Intersects(POLYGON ((137.7729 13.1342, 137.7729 23.8860, 153.7491 23.8860, 153.7491 13.1342, 137.7729 13.1342)))" AND productType:S2MSI1C AND beginPosition:[2020-08-08T00:00:00.000Z TO *] AND endPosition:[* TO 2020-08-16T00:00:00.000Z]"'  # noqa
+        self.onda_url_count = (
+            'https://catalogue.onda-dias.eu/dias-catalogue/Products/$count?$search="footprint:"'
+            "Intersects(POLYGON ((137.7729 13.1342, 137.7729 23.8860, 153.7491 23.8860, 153.7491 13.1342, "
+            '137.7729 13.1342)))" AND productType:S2MSI1C AND beginPosition:[2020-08-08T00:00:00.000Z TO *] '
+            'AND endPosition:[* TO 2020-08-16T00:00:00.000Z] AND foo:bar"'
+        )
+
         self.onda_products_count = 47
 
     @mock.patch("eodag.plugins.search.qssearch.requests.get", autospec=True)
@@ -414,12 +420,19 @@ class TestSearchPluginODataV4Search(BaseSearchPluginTest):
             page=1,
             items_per_page=2,
             auth=self.onda_auth_plugin,
+            # custom query argument that must be mapped using discovery_metata.search_param
+            foo="bar",
             **self.search_criteria_s2_msi_l1c
         )
 
         # Specific expected results
         number_of_products = 2
-        onda_url_search = 'https://catalogue.onda-dias.eu/dias-catalogue/Products?productType=S2MSI1C&$format=json&$search="footprint:"Intersects(POLYGON ((137.7729 13.1342, 137.7729 23.8860, 153.7491 23.8860, 153.7491 13.1342, 137.7729 13.1342)))" AND productType:S2MSI1C AND beginPosition:[2020-08-08T00:00:00.000Z TO *] AND endPosition:[* TO 2020-08-16T00:00:00.000Z]"&$top=2&$skip=0'  # noqa
+        onda_url_search = (
+            'https://catalogue.onda-dias.eu/dias-catalogue/Products?$format=json&$search="'
+            'footprint:"Intersects(POLYGON ((137.7729 13.1342, 137.7729 23.8860, 153.7491 23.8860, 153.7491 13.1342, '
+            '137.7729 13.1342)))" AND productType:S2MSI1C AND beginPosition:[2020-08-08T00:00:00.000Z TO *] '
+            'AND endPosition:[* TO 2020-08-16T00:00:00.000Z] AND foo:bar"&$top=2&$skip=0&$expand=Metadata'
+        )
 
         mock__request.assert_any_call(
             mock.ANY,