Repository: CS-SI/eodag

Issue:
Search for all the products matching some criteria with a given provider
We have had a lot of discussions on a private GitLab instance before we moved to GitHub about how to search/download all the products matching some criteria.

The problems we observed with the current API were:
* The method `search` returns a tuple of `(SearchResult, int)`. The first item contains some products, in most cases 20 if the user hasn't defined `items_per_page` in the search (20 being the default value) or less than 20 if less products match the criteria. The second item is the total number of products matching the criteria, it is obtained after sending a count request (or equivalent) to the provider. This number may in some cases just be an estimate, usually lower than the correct value (this is true for peps at the time of writing). So what happens is that the user actually gets the results from the first page only, but it's not so easy to understand (https://github.com/CS-SI/eodag/issues/149)
* We have the method `download_all` that does what it says, it downloads all the products contained in the `SearchResult` it takes as an argument. However, some users would expect that it just downloads all the products matching the criteria that were used to search them in the first place.
* There is no clear way or example about how to search (and download) for all the products matching some criteria. The current way would be to execute a `search` by incrementing the page value given as a parameter. The issue with this approach is that it would each time send a count request (whether it's a proper request or not depends on the underlying search plugin implementation, but in most cases it's currently the case), while this doesn't seem necessary. Or by increasing `items_per_page` to a very high value, which may be too high (providers have their own limit) and result in potentially an obscure error.

We have discussed different approaches:
1. Add some superpower to `download_all` so that it could download more than the products it would be given. The idea was to store in the `SearchResult` all the information required to run the same search, and as a consequence to run the search on the next page. The next products found would then have been downloaded. And so on until exhaustion of the results. It would have been used as follows: `products, _ = dag.search(**criteria); paths = dag.download_all(products, exhaust=True)`
2. Add a `search_all` method that would collect all the products matching some criteria.
  2.1. By adding to `SearchResult` the possibility to trigger a search for the next page (same idea as with `download_all`)
  2.2. By adding a `search_iter_page` method to `EODataAccessGateway` that would iterate over the pages of a search, and using that method internally to implement `search_all`.

The approach *2.2.* was chosen to be implemented (which doesn't necessarily mean the other approaches can't or won't be implemented if they prove to be required):
* Just adding more power to `download_all` felt like it was not sufficient. Since some providers do not return the exact count of products available in a `search`, it would have been impossible for these providers to know beforehand how many products  `download_all` would download. This feels a a little inappropriate.
* `search_all` happens to mirror `download_all` quite well, the usage of `products = dag.search_all(**criteria); paths = dag.download_all(products)` feels quite natural
* Exposing a method `search_iter_page` will help us to document and explain that (most) searches are paginated. This is not obvious to every user while this is important to understand how eodag works.

Discussion:
sbrunato: For `astraea_eod`, STAC API pagination only needs `item["links"][]["rel"] == "next"`. `page` parameter is only optional, see https://github.com/radiantearth/stac-api-spec/tree/master/item-search#paging.

For STAC providers, we should this `next` entry by default instead of `page`+`limit`.

This could be done by adding a new parameter in `providers.yml` / `stac_provider.yml`:
```yml
  pagination:
    next_page_url_key_path: '$.links[?(@.rel="next")].href'
```
And use `jsonpath_ng.ext.parse` instead of `jsonpath_ng.parse`, see https://github.com/h2non/jsonpath-ng/issues/8
maximlt: @sbrunato I have implemented a mechanism to retrieve the next page when a provider doesn't implement "page".

In `search_iter_page` a search plugin is introspected to check whether it has a `pagination.next_page_url_key_path` key. If it does, it is assumed that it doesn't implement "page", and a kwarg `set_next_page_url` is passed down to `.query`. This is handled within `QueryStringSearch.do_search` where the response (JSON only right now) is parsed to get the "next" URL. It replaces the configuration key `pagination.next_page_url_tpl`. So the next call to `.query` will use that URL and will update it to get the next page.

I have added comments since this logic is quite special. Maybe we'll come up with a better way in the future.
maximlt: @sbrunato I've made the changes as we agreed:
* `search_iter_page` doesn't propagate a boolean to trigger to "next" search through the plugin infrastructure down to `do_search`. Instead `do_search` takes care of setting an attribute `next_page_url` if the pagination config has the key `next_page_url_key_path`.
* `search_iter_page` uses that URL after the first iteration if available. It uses it by replacing `search_plugin.config.pagination["next_page_url_tpl"]`
* `search_plugin.config.pagination["next_page_url_tpl"]` and `search_plugin.next_page_url` are both reset after each iteration. The generator holds the `next_page_url` value in its internal state. This has the advantage that the search plugin is in a clean state after `next(dag.search_iter_page(**search_criteria))`. I thought it was quite important that `search_plugin.next_page_url` was reset every time, I didn't want a URL from a previous search to mess up with a new search (different criteria).

Patch:
diff --git a/eodag/api/core.py b/eodag/api/core.py
index 4655c3e3..6b84203c 100644
--- a/eodag/api/core.py
+++ b/eodag/api/core.py
@@ -59,6 +59,10 @@ logger = logging.getLogger("eodag.core")
 # pagination defaults
 DEFAULT_PAGE = 1
 DEFAULT_ITEMS_PER_PAGE = 20
+# Default maximum number of items per page requested by search_all. 50 instead of
+# 20 (DEFAULT_ITEMS_PER_PAGE) to increase it to the known and currentminimum
+# value (mundi)
+DEFAULT_MAX_ITEMS_PER_PAGE = 50
 
 
 class EODataAccessGateway(object):
@@ -243,7 +247,9 @@ class EODataAccessGateway(object):
         :type provider: str
         """
         if provider not in self.available_providers():
-            raise UnsupportedProvider("This provider is not recognised by eodag")
+            raise UnsupportedProvider(
+                f"This provider is not recognised by eodag: {provider}"
+            )
         preferred_provider, max_priority = self.get_preferred_provider()
         if preferred_provider != provider:
             new_priority = max_priority + 1
@@ -391,22 +397,6 @@ class EODataAccessGateway(object):
     def guess_product_type(self, **kwargs):
         """Find the eodag product type code that best matches a set of search params
 
-        >>> from eodag import EODataAccessGateway
-        >>> dag = EODataAccessGateway()
-        >>> dag.guess_product_type(
-        ...     instrument="MSI",
-        ...     platform="SENTINEL2",
-        ...     platformSerialIdentifier="S2A",
-        ... ) # doctest: +NORMALIZE_WHITESPACE
-        ['S2_MSI_L1C', 'S2_MSI_L2A', 'S2_MSI_L2A_MAJA', 'S2_MSI_L2B_MAJA_SNOW',
-            'S2_MSI_L2B_MAJA_WATER', 'S2_MSI_L3A_WASP']
-        >>> import eodag.utils.exceptions
-        >>> try:
-        ...     dag.guess_product_type()
-        ...     raise AssertionError(u"NoMatchingProductType exception not raised")
-        ... except eodag.utils.exceptions.NoMatchingProductType:
-        ...     pass
-
         :param kwargs: A set of search parameters as keywords arguments
         :return: The best match for the given parameters
         :rtype: str
@@ -455,7 +445,7 @@ class EODataAccessGateway(object):
         end=None,
         geom=None,
         locations=None,
-        **kwargs
+        **kwargs,
     ):
         """Look for products matching criteria on known providers.
 
@@ -557,6 +547,295 @@ class EODataAccessGateway(object):
             return a list as a result of their processing. This requirement is
             enforced here.
         """
+        search_kwargs = self._prepare_search(
+            start=start, end=end, geom=geom, locations=locations, **kwargs
+        )
+        if search_kwargs.get("id"):
+            provider = search_kwargs.get("provider")
+            return self._search_by_id(search_kwargs["id"], provider)
+        search_plugin = search_kwargs.pop("search_plugin")
+        search_kwargs.update(
+            page=page,
+            items_per_page=items_per_page,
+        )
+        return self._do_search(
+            search_plugin, count=True, raise_errors=raise_errors, **search_kwargs
+        )
+
+    def search_iter_page(
+        self,
+        items_per_page=DEFAULT_ITEMS_PER_PAGE,
+        start=None,
+        end=None,
+        geom=None,
+        locations=None,
+        **kwargs,
+    ):
+        """Iterate over the pages of a products search.
+
+        :param items_per_page: The number of results requested per page (default: 20)
+        :type items_per_page: int
+        :param start: Start sensing UTC time in iso format
+        :type start: str
+        :param end: End sensing UTC time in iso format
+        :type end: str
+        :param geom: Search area that can be defined in different ways:
+
+                    * with a Shapely geometry object:
+                      ``class:`shapely.geometry.base.BaseGeometry```
+                    * with a bounding box (dict with keys: "lonmin", "latmin", "lonmax", "latmax"):
+                      ``dict.fromkeys(["lonmin", "latmin", "lonmax", "latmax"])``
+                    * with a bounding box as list of float:
+                      ``[lonmin, latmin, lonmax, latmax]``
+                    * with a WKT str
+
+        :type geom: Union[str, dict, shapely.geometry.base.BaseGeometry]
+        :param locations: Location filtering by name using locations configuration
+                          ``{"<location_name>"="<attr_regex>"}``. For example, ``{"country"="PA."}`` will use
+                          the geometry of the features having the property ISO3 starting with
+                          'PA' such as Panama and Pakistan in the shapefile configured with
+                          name=country and attr=ISO3
+        :type locations: dict
+        :param dict kwargs: some other criteria that will be used to do the search,
+                            using paramaters compatibles with the provider
+        :returns: An iterator that yields page per page a collection of EO products
+                  matching the criteria
+        :rtype: Iterator[:class:`~eodag.api.search_result.SearchResult`]
+
+        .. versionadded::
+            2.2.0
+        """
+        search_kwargs = self._prepare_search(
+            start=start, end=end, geom=geom, locations=locations, **kwargs
+        )
+        search_plugin = search_kwargs.pop("search_plugin")
+        iteration = 1
+        # Store the search plugin config pagination.next_page_url_tpl to reset it later
+        # since it might be modified if the next_page_url mechanism is used by the
+        # plugin.
+        pagination_config = getattr(search_plugin.config, "pagination", {})
+        prev_next_page_url_tpl = pagination_config.get("next_page_url_tpl")
+        # Page has to be set to a value even if use_next is True, this is required
+        # internally by the search plugin (see collect_search_urls)
+        search_kwargs.update(
+            page=1,
+            items_per_page=items_per_page,
+        )
+        prev_product = None
+        next_page_url = None
+        while True:
+            if iteration > 1 and next_page_url:
+                pagination_config["next_page_url_tpl"] = next_page_url
+            logger.debug("Iterate over pages: search page %s", iteration)
+            try:
+                products, _ = self._do_search(
+                    search_plugin, count=False, raise_errors=True, **search_kwargs
+                )
+            finally:
+                # we don't want that next(search_iter_page(...)) modifies the plugin
+                # indefinitely. So we reset after each request, but before the generator
+                # yields, the attr next_page_url (to None) and
+                # config.pagination["next_page_url_tpl"] (to its original value).
+                next_page_url = getattr(search_plugin, "next_page_url", None)
+                if next_page_url:
+                    search_plugin.next_page_url = None
+                    if prev_next_page_url_tpl:
+                        search_plugin.config.pagination[
+                            "next_page_url_tpl"
+                        ] = prev_next_page_url_tpl
+            if len(products) > 0:
+                # The first products between two iterations are compared. If they
+                # are actually the same product, it means the iteration failed at
+                # progressing for some reason. This is implemented as a workaround
+                # to some search plugins/providers not handling pagination.
+                product = products[0]
+                if (
+                    prev_product
+                    and product.properties["id"] == prev_product.properties["id"]
+                    and product.provider == prev_product.provider
+                ):
+                    logger.warning(
+                        "Iterate over pages: stop iterating since the next page "
+                        "appears to have the same products as in the previous one. "
+                        "This provider may not implement pagination.",
+                    )
+                    last_page_with_products = iteration - 1
+                    break
+                yield products
+                prev_product = product
+                # Prevent a last search if the current one returned less than the
+                # maximum number of items asked for.
+                if len(products) < items_per_page:
+                    last_page_with_products = iteration
+                    break
+            else:
+                last_page_with_products = iteration - 1
+                break
+            iteration += 1
+            search_kwargs["page"] = iteration
+        logger.debug(
+            "Iterate over pages: last products found on page %s",
+            last_page_with_products,
+        )
+
+    def search_all(
+        self,
+        items_per_page=None,
+        start=None,
+        end=None,
+        geom=None,
+        locations=None,
+        **kwargs,
+    ):
+        """Search and return all the products matching the search criteria.
+
+        It iterates over the pages of a search query and collects all the returned
+        products into a single :class:`~eodag.api.search_result.SearchResult` instance.
+
+        :param items_per_page: (optional) The number of results requested internally per
+                               page. The maximum number of items than can be requested
+                               at once to a provider has been configured in EODAG for
+                               some of them. If items_per_page is None and this number
+                               is available for the searched provider, it is used to
+                               limit the number of requests made. This should also
+                               reduce the time required to collect all the products
+                               matching the search criteria. If this number is not
+                               available, a default value of 50 is used instead.
+                               items_per_page can also be set to any arbitrary value.
+        :type items_per_page: int
+        :param start: Start sensing UTC time in iso format
+        :type start: str
+        :param end: End sensing UTC time in iso format
+        :type end: str
+        :param geom: Search area that can be defined in different ways:
+
+                    * with a Shapely geometry object:
+                      ``class:`shapely.geometry.base.BaseGeometry```
+                    * with a bounding box (dict with keys: "lonmin", "latmin", "lonmax", "latmax"):
+                      ``dict.fromkeys(["lonmin", "latmin", "lonmax", "latmax"])``
+                    * with a bounding box as list of float:
+                      ``[lonmin, latmin, lonmax, latmax]``
+                    * with a WKT str
+
+        :type geom: Union[str, dict, shapely.geometry.base.BaseGeometry]
+        :param locations: Location filtering by name using locations configuration
+                          ``{"<location_name>"="<attr_regex>"}``. For example, ``{"country"="PA."}`` will use
+                          the geometry of the features having the property ISO3 starting with
+                          'PA' such as Panama and Pakistan in the shapefile configured with
+                          name=country and attr=ISO3
+        :type locations: dict
+        :param dict kwargs: some other criteria that will be used to do the search,
+                            using paramaters compatibles with the provider
+        :returns: An iterator that yields page per page a collection of EO products
+                  matching the criteria
+        :rtype: Iterator[:class:`~eodag.api.search_result.SearchResult`]
+
+        .. versionadded::
+            2.2.0
+        """
+        # Prepare the search just to get the search plugin and the maximized value
+        # of items_per_page if defined for the provider used.
+        search_kwargs = self._prepare_search(
+            start=start, end=end, geom=geom, locations=locations, **kwargs
+        )
+        search_plugin = search_kwargs["search_plugin"]
+        if items_per_page is None:
+            items_per_page = search_plugin.config.pagination.get(
+                "max_items_per_page", DEFAULT_MAX_ITEMS_PER_PAGE
+            )
+        logger.debug(
+            "Searching for all the products with provider %s and a maximum of %s "
+            "items per page.",
+            search_plugin.provider,
+            items_per_page,
+        )
+        all_results = SearchResult([])
+        for page_results in self.search_iter_page(
+            items_per_page=items_per_page,
+            start=start,
+            end=end,
+            geom=geom,
+            locations=locations,
+            **kwargs,
+        ):
+            all_results.data.extend(page_results.data)
+        logger.info(
+            "Found %s result(s) on provider '%s'",
+            len(all_results),
+            search_plugin.provider,
+        )
+        return all_results
+
+    def _search_by_id(self, uid, provider=None):
+        """Internal method that enables searching a product by its id.
+
+        Keeps requesting providers until a result matching the id is supplied. The
+        search plugins should be developed in the way that enable them to handle the
+        support of a search by id by the providers. The providers are requested one by
+        one, in the order defined by their priorities. Be aware that because of that,
+        the search can be slow, if the priority order is such that the provider that
+        contains the requested product has the lowest priority. However, you can always
+        speed up a little the search by passing the name of the provider on which to
+        perform the search, if this information is available
+
+        :param uid: The uid of the EO product
+        :type uid: str
+        :param provider: (optional) The provider on which to search the product.
+                         This may be useful for performance reasons when the user
+                         knows this product is available on the given provider
+        :type provider: str
+        :returns: A search result with one EO product or None at all, and the number
+                  of EO products retrieved (0 or 1)
+        :rtype: tuple(:class:`~eodag.api.search_result.SearchResult`, int)
+
+        .. versionadded:: 1.0
+        """
+        for plugin in self._plugins_manager.get_search_plugins(provider=provider):
+            logger.info(
+                "Searching product with id '%s' on provider: %s", uid, plugin.provider
+            )
+            logger.debug("Using plugin class for search: %s", plugin.__class__.__name__)
+            auth = self._plugins_manager.get_auth_plugin(plugin.provider)
+            results, _ = self._do_search(plugin, auth=auth, id=uid)
+            if len(results) == 1:
+                return results, 1
+        return SearchResult([]), 0
+
+    def _prepare_search(
+        self, start=None, end=None, geom=None, locations=None, **kwargs
+    ):
+        """Internal method to prepare the search kwargs and get the search
+        and auth plugins.
+
+        Product query:
+          * By id (plus optional 'provider')
+          * By search params:
+            * productType query:
+              * By product type (e.g. 'S2_MSI_L1C')
+              * By params (e.g. 'platform'), see guess_product_type
+            * dates: 'start' and/or 'end'
+            * geometry: 'geom' or 'bbox' or 'box'
+            * search locations
+            * TODO: better expose cloudCover
+            * other search params are passed to Searchplugin.query()
+
+        :param start: Start sensing UTC time in iso format
+        :type start: str
+        :param end: End sensing UTC time in iso format
+        :type end: str
+        :param geom: Search area that can be defined in different ways (see search)
+        :type geom: Union[str, dict, shapely.geometry.base.BaseGeometry]
+        :param locations: Location filtering by name using locations configuration
+        :type locations: dict
+        :param dict kwargs: Some other criteria
+                            * id and/or a provider for a search by
+                            * search criteria to guess the product type
+                            * other criteria compatible with the provider
+        :returns: The prepared kwargs to make a query.
+        :rtype: dict
+
+        .. versionadded:: 2.2
+        """
         product_type = kwargs.get("productType", None)
         if product_type is None:
             try:
@@ -570,14 +849,18 @@ class EODataAccessGateway(object):
                         "No product type could be guessed with provided arguments"
                     )
                 else:
-                    provider = kwargs.get("provider", None)
-                    return self._search_by_id(kwargs["id"], provider=provider)
+                    return kwargs
 
         kwargs["productType"] = product_type
         if start is not None:
             kwargs["startTimeFromAscendingNode"] = start
         if end is not None:
             kwargs["completionTimeFromAscendingNode"] = end
+        if "box" in kwargs or "bbox" in kwargs:
+            logger.warning(
+                "'box' or 'bbox' parameters are only supported for backwards "
+                " compatibility reasons. Usage of 'geom' is recommended."
+            )
         if geom is not None:
             kwargs["geometry"] = geom
         box = kwargs.pop("box", None)
@@ -593,95 +876,73 @@ class EODataAccessGateway(object):
             kwargs.pop(arg, None)
         del kwargs["locations"]
 
-        plugin = next(
+        search_plugin = next(
             self._plugins_manager.get_search_plugins(product_type=product_type)
         )
         logger.info(
-            "Searching product type '%s' on provider: %s", product_type, plugin.provider
+            "Searching product type '%s' on provider: %s",
+            product_type,
+            search_plugin.provider,
         )
-        # add product_types_config to plugin config
+        # Add product_types_config to plugin config. This dict contains product
+        # type metadata that will also be stored in each product's properties.
         try:
-            plugin.config.product_type_config = dict(
+            search_plugin.config.product_type_config = dict(
                 [
                     p
-                    for p in self.list_product_types(plugin.provider)
+                    for p in self.list_product_types(search_plugin.provider)
                     if p["ID"] == product_type
                 ][0],
-                **{"productType": product_type}
+                **{"productType": product_type},
             )
+        # If the product isn't in the catalog, it's a generic product type.
         except IndexError:
-            plugin.config.product_type_config = dict(
+            search_plugin.config.product_type_config = dict(
                 [
                     p
-                    for p in self.list_product_types(plugin.provider)
+                    for p in self.list_product_types(search_plugin.provider)
                     if p["ID"] == GENERIC_PRODUCT_TYPE
                 ][0],
-                **{"productType": product_type}
+                **{"productType": product_type},
             )
-        plugin.config.product_type_config.pop("ID", None)
+        # Remove the ID since this is equal to productType.
+        search_plugin.config.product_type_config.pop("ID", None)
 
-        logger.debug("Using plugin class for search: %s", plugin.__class__.__name__)
-        auth = self._plugins_manager.get_auth_plugin(plugin.provider)
-        return self._do_search(
-            plugin,
-            auth=auth,
-            page=page,
-            items_per_page=items_per_page,
-            raise_errors=raise_errors,
-            **kwargs
+        logger.debug(
+            "Using plugin class for search: %s", search_plugin.__class__.__name__
         )
+        auth_plugin = self._plugins_manager.get_auth_plugin(search_plugin.provider)
 
-    def _search_by_id(self, uid, provider=None):
-        """Internal method that enables searching a product by its id.
-
-        Keeps requesting providers until a result matching the id is supplied. The
-        search plugins should be developed in the way that enable them to handle the
-        support of a search by id by the providers. The providers are requested one by
-        one, in the order defined by their priorities. Be aware that because of that,
-        the search can be slow, if the priority order is such that the provider that
-        contains the requested product has the lowest priority. However, you can always
-        speed up a little the search by passing the name of the provider on which to
-        perform the search, if this information is available
+        return dict(search_plugin=search_plugin, auth=auth_plugin, **kwargs)
 
-        :param uid: The uid of the EO product
-        :type uid: str
-        :param provider: (optional) The provider on which to search the product.
-                         This may be useful for performance reasons when the user
-                         knows this product is available on the given provider
-        :type provider: str
-        :returns: A search result with one EO product or None at all, and the number
-                  of EO products retrieved (0 or 1)
-        :rtype: tuple(:class:`~eodag.api.search_result.SearchResult`, int)
-
-        .. versionadded:: 1.0
-        """
-        for plugin in self._plugins_manager.get_search_plugins(provider=provider):
-            logger.info(
-                "Searching product with id '%s' on provider: %s", uid, plugin.provider
-            )
-            logger.debug("Using plugin class for search: %s", plugin.__class__.__name__)
-            auth = self._plugins_manager.get_auth_plugin(plugin.provider)
-            results, _ = self._do_search(plugin, auth=auth, id=uid)
-            if len(results) == 1:
-                return results, 1
-        return SearchResult([]), 0
-
-    def _do_search(self, search_plugin, **kwargs):
+    def _do_search(self, search_plugin, count=True, raise_errors=False, **kwargs):
         """Internal method that performs a search on a given provider.
 
+        :param search_plugin: A search plugin
+        :type search_plugin: eodag.plugins.base.Search
+        :param count: Whether to run a query with a count request or not (default: True)
+        :type count: bool
+        :param raise_errors:  When an error occurs when searching, if this is set to
+                              True, the error is raised (default: False)
+        :type raise_errors: bool
+        :param dict kwargs: some other criteria that will be used to do the search
+        :returns: A collection of EO products matching the criteria and the total
+                  number of results found if count is True else None
+        :rtype: tuple(:class:`~eodag.api.search_result.SearchResult`, int or None)
+
         .. versionadded:: 1.0
         """
         results = SearchResult([])
         total_results = 0
         try:
-            res, nb_res = search_plugin.query(**kwargs)
+            res, nb_res = search_plugin.query(count=count, **kwargs)
 
             # Only do the pagination computations when it makes sense. For example,
             # for a search by id, we can reasonably guess that the provider will return
             # At most 1 product, so we don't need such a thing as pagination
             page = kwargs.get("page")
             items_per_page = kwargs.get("items_per_page")
-            if page and items_per_page:
+            if page and items_per_page and count:
                 # Take into account the fact that a provider may not return the count of
                 # products (in that case, fallback to using the length of the results it
                 # returned and the page requested. As an example, check the result of
@@ -739,31 +1000,34 @@ class EODataAccessGateway(object):
                     )
 
             results.extend(res)
-            total_results += nb_res
-            logger.info(
-                "Found %s result(s) on provider '%s'", nb_res, search_plugin.provider
-            )
-            # Hitting for instance
-            # https://theia.cnes.fr/atdistrib/resto2/api/collections/SENTINEL2/
-            #   search.json?startDate=2019-03-01&completionDate=2019-06-15
-            #   &processingLevel=LEVEL2A&maxRecords=1&page=1
-            # returns a number (properties.totalResults) that is the number of
-            # products in the collection (here SENTINEL2) instead of the estimated
-            # total number of products matching the search criteria (start/end date).
-            # Remove this warning when this is fixed upstream by THEIA.
-            if search_plugin.provider == "theia":
-                logger.warning(
-                    "Results found on provider 'theia' is the total number of products "
-                    "available in the searched collection (e.g. SENTINEL2) instead of "
-                    "the total number of products matching the search criteria"
+            total_results = None if nb_res is None else total_results + nb_res
+            if count:
+                logger.info(
+                    "Found %s result(s) on provider '%s'",
+                    nb_res,
+                    search_plugin.provider,
                 )
+                # Hitting for instance
+                # https://theia.cnes.fr/atdistrib/resto2/api/collections/SENTINEL2/
+                #   search.json?startDate=2019-03-01&completionDate=2019-06-15
+                #   &processingLevel=LEVEL2A&maxRecords=1&page=1
+                # returns a number (properties.totalResults) that is the number of
+                # products in the collection (here SENTINEL2) instead of the estimated
+                # total number of products matching the search criteria (start/end date).
+                # Remove this warning when this is fixed upstream by THEIA.
+                if search_plugin.provider == "theia":
+                    logger.warning(
+                        "Results found on provider 'theia' is the total number of products "
+                        "available in the searched collection (e.g. SENTINEL2) instead of "
+                        "the total number of products matching the search criteria"
+                    )
         except Exception:
             logger.info(
                 "No result from provider '%s' due to an error during search. Raise "
                 "verbosity of log messages for details",
                 search_plugin.provider,
             )
-            if kwargs.get("raise_errors"):
+            if raise_errors:
                 # Raise the error, letting the application wrapping eodag know that
                 # something went bad. This way it will be able to decide what to do next
                 raise
@@ -827,7 +1091,7 @@ class EODataAccessGateway(object):
         progress_callback=None,
         wait=DEFAULT_DOWNLOAD_WAIT,
         timeout=DEFAULT_DOWNLOAD_TIMEOUT,
-        **kwargs
+        **kwargs,
     ):
         """Download all products resulting from a search.
 
@@ -865,7 +1129,7 @@ class EODataAccessGateway(object):
                 progress_callback=progress_callback,
                 wait=wait,
                 timeout=timeout,
-                **kwargs
+                **kwargs,
             )
             # close progress_bar when finished
             if hasattr(progress_callback, "pb") and hasattr(
@@ -931,7 +1195,7 @@ class EODataAccessGateway(object):
         provider=None,
         productType=None,
         timeout=HTTP_REQ_TIMEOUT,
-        **kwargs
+        **kwargs,
     ):
         """Loads STAC items from a geojson file / STAC catalog or collection, and convert to SearchResult.
 
@@ -1000,7 +1264,7 @@ class EODataAccessGateway(object):
         progress_callback=None,
         wait=DEFAULT_DOWNLOAD_WAIT,
         timeout=DEFAULT_DOWNLOAD_TIMEOUT,
-        **kwargs
+        **kwargs,
     ):
         """Download a single product.
 
diff --git a/eodag/plugins/search/qssearch.py b/eodag/plugins/search/qssearch.py
index 074ed5cb..b238440a 100644
--- a/eodag/plugins/search/qssearch.py
+++ b/eodag/plugins/search/qssearch.py
@@ -92,6 +92,9 @@ class QueryStringSearch(Search):
           - *count_endpoint*: (optional) The endpoint for counting the number of items
             satisfying a request
 
+            *next_page_url_key_path: (optional) A JSONPATH expression used to retrieve
+            the URL of the next page in the response of the current page.
+
         - **free_text_search_operations**: (optional) A tree structure of the form::
 
             <search-param>:     # e.g: $search
@@ -153,9 +156,19 @@ class QueryStringSearch(Search):
         self.search_urls = []
         self.query_params = dict()
         self.query_string = ""
+        self.next_page_url = None
 
     def query(self, items_per_page=None, page=None, count=True, **kwargs):
-        """Perform a search on an OpenSearch-like interface"""
+        """Perform a search on an OpenSearch-like interface
+
+        :param page: The page number to retur (default: 1)
+        :type page: int
+        :param items_per_page: The number of results that must appear in one single
+                               page
+        :type items_per_page: int
+        :param count:  To trigger a count request (default: True)
+        :type count: bool
+        """
         product_type = kwargs.get("productType", None)
         if product_type == GENERIC_PRODUCT_TYPE:
             logger.warning(
@@ -417,6 +430,9 @@ class QueryStringSearch(Search):
             except RequestError:
                 raise StopIteration
             else:
+                next_page_url_key_path = self.config.pagination.get(
+                    "next_page_url_key_path"
+                )
                 if self.config.result_type == "xml":
                     root_node = etree.fromstring(response.content)
                     namespaces = {k or "ns": v for k, v in root_node.nsmap.items()}
@@ -426,8 +442,24 @@ class QueryStringSearch(Search):
                             self.config.results_entry, namespaces=namespaces
                         )
                     ]
+                    if next_page_url_key_path:
+                        raise NotImplementedError(
+                            "Setting the next page url from an XML response has not "
+                            "been implemented yet"
+                        )
                 else:
-                    result = response.json().get(self.config.results_entry, [])
+                    resp_as_json = response.json()
+                    if next_page_url_key_path:
+                        path_parsed = parse(next_page_url_key_path)
+                        try:
+                            self.next_page_url = path_parsed.find(resp_as_json)[0].value
+                            logger.debug(
+                                "Next page URL collected and set for the next search",
+                            )
+                        except IndexError:
+                            logger.debug("Next page URL could not be collected")
+
+                    result = resp_as_json.get(self.config.results_entry, [])
                 if getattr(self.config, "merge_responses", False):
                     results = (
                         [dict(r, **result[i]) for i, r in enumerate(results)]
diff --git a/eodag/resources/providers.yml b/eodag/resources/providers.yml
index c034296e..9906327a 100644
--- a/eodag/resources/providers.yml
+++ b/eodag/resources/providers.yml
@@ -504,6 +504,8 @@
     pagination:
       next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&page={page}'
       total_items_nb_key_path: '$.properties.totalResults'
+      # 2019/03/19: Returns a 400 error code if greater than 500.
+      max_items_per_page: 500
     discover_metadata:
       auto_discovery: true
       metadata_pattern: '^(?!collection)[a-zA-Z0-9_]+$'
@@ -676,6 +678,8 @@
     pagination:
       next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&page={page}'
       total_items_nb_key_path: '$.properties.totalResults'
+      # 2019/03/19: 500 is the max, no error if greater
+      max_items_per_page: 500
     discover_metadata:
       auto_discovery: true
       metadata_pattern: '^(?!collection)[a-zA-Z0-9_]+$'
@@ -866,6 +870,9 @@
     pagination:
       next_page_url_tpl: '{url}?{search}&size={items_per_page}&from={skip}'
       total_items_nb_key_path: '$.totalnb'
+      # 2019/03/19: It's not possible to get more than 10_000 products
+      # Returns a 400 Bad Request if more products are requested.
+      max_items_per_page: 10_000
     results_entry: 'hits'
     discover_metadata:
       auto_discovery: true
@@ -953,6 +960,8 @@
     pagination:
       next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&page={page}'
       total_items_nb_key_path: '$.properties.totalResults'
+      # 2019/03/19: 2000 is the max, 400 error if greater
+      max_items_per_page: 2_000
     discover_metadata:
       auto_discovery: true
       metadata_pattern: '^(?!collection)[a-zA-Z0-9]+$'
@@ -1106,6 +1115,8 @@
     pagination:
       next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&startIndex={skip_base_1}'
       total_items_nb_key_path: '//os:totalResults/text()'
+      # 2019/03/19: 50 is the max, no error if greater
+      max_items_per_page: 50
     discover_metadata:
       auto_discovery: true
       metadata_pattern: '^(?!collection)[a-zA-Z0-9]+$'
@@ -1328,6 +1339,8 @@
     pagination:
       count_endpoint: 'https://catalogue.onda-dias.eu/dias-catalogue/Products/$count'
       next_page_url_tpl: '{url}?{search}&$top={items_per_page}&$skip={skip}'
+      # 2019/03/19: 2000 is the max, if greater 200 response but contains an error message
+      max_items_per_page: 2_000
     results_entry: 'value'
     literal_search_params:
       $format: json
@@ -1463,6 +1476,13 @@
   search: !plugin
     type: StacSearch
     api_endpoint: https://eod-catalog-svc-prod.astraea.earth/search
+    pagination:
+      # 2019/03/19: The docs (https://eod-catalog-svc-prod.astraea.earth/api.html#operation/getSearchSTAC)
+      # say the max is 10_000. In practice 1_000 products are returned if more are asked (even greater
+      # than 10_000), without any error.
+      # This provider doesn't implement any pagination, let's just try to get the maximum number of
+      # products available at once then, so we stick to 10_000.
+      max_items_per_page: 10_000
   products:
     S2_MSI_L1C:
       productType: sentinel2_l1c
@@ -1496,6 +1516,10 @@
     api_endpoint: https://landsatlook.usgs.gov/sat-api/collections/{collection}/items
     pagination:
       total_items_nb_key_path: '$.meta.found'
+      # 2019/03/19: no more than 10_000 (if greater, returns a 500 error code)
+      # but in practive if an Internal Server Error is returned for more than
+      # about 500 products.
+      max_items_per_page: 500
     metadata_mapping:
       productType: '$.collection'
       completionTimeFromAscendingNode:
diff --git a/eodag/resources/stac_provider.yml b/eodag/resources/stac_provider.yml
index e3c8848c..b6f2ed19 100644
--- a/eodag/resources/stac_provider.yml
+++ b/eodag/resources/stac_provider.yml
@@ -20,8 +20,7 @@ search:
   pagination:
     next_page_url_tpl: '{url}?{search}&limit={items_per_page}&page={page}'
     total_items_nb_key_path: '$.context.matched'
-    pagination:
-      next_page_url_key_path: '$.links[?(@.rel="next")].href'
+    next_page_url_key_path: '$.links[?(@.rel="next")].href'
   discover_metadata:
     auto_discovery: true
     metadata_pattern: '^[a-zA-Z0-9_:-]+$'


Test patch:
diff --git a/tests/context.py b/tests/context.py
index bd969304..83d9354e 100644
--- a/tests/context.py
+++ b/tests/context.py
@@ -25,7 +25,7 @@ import sys
 sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
 
 from eodag import EODataAccessGateway, api, config
-from eodag.api.core import DEFAULT_ITEMS_PER_PAGE
+from eodag.api.core import DEFAULT_ITEMS_PER_PAGE, DEFAULT_MAX_ITEMS_PER_PAGE
 from eodag.api.product import EOProduct
 from eodag.api.product.drivers import DRIVERS
 from eodag.api.product.drivers.base import NoDriver
@@ -49,6 +49,8 @@ from eodag.utils.exceptions import (
     AuthenticationError,
     DownloadError,
     MisconfiguredError,
+    NoMatchingProductType,
+    PluginImplementationError,
     UnsupportedDatasetAddressScheme,
     UnsupportedProvider,
     ValidationError,
diff --git a/tests/test_end_to_end.py b/tests/test_end_to_end.py
index 08c2521a..451996a2 100644
--- a/tests/test_end_to_end.py
+++ b/tests/test_end_to_end.py
@@ -140,17 +140,17 @@ class EndToEndBase(unittest.TestCase):
         - Return one product to be downloaded
         """
         search_criteria = {
+            "productType": product_type,
             "startTimeFromAscendingNode": start,
             "completionTimeFromAscendingNode": end,
             "geom": geom,
         }
+        if items_per_page:
+            search_criteria["items_per_page"] = items_per_page
+        if page:
+            search_criteria["page"] = page
         self.eodag.set_preferred_provider(provider)
-        results, nb_results = self.eodag.search(
-            productType=product_type,
-            page=page,
-            items_per_page=items_per_page,
-            **search_criteria
-        )
+        results, nb_results = self.eodag.search(**search_criteria)
         if offline:
             results = [
                 prod
@@ -165,6 +165,38 @@ class EndToEndBase(unittest.TestCase):
         else:
             return results
 
+    def execute_search_all(
+        self,
+        provider,
+        product_type,
+        start,
+        end,
+        geom,
+        items_per_page=None,
+        check_products=True,
+    ):
+        """Search all the products on provider:
+
+        - First set the preferred provider as the one given in parameter
+        - Then do the search_all
+        - Then ensure that at least the first result originates from the provider
+        - Return all the products
+        """
+        search_criteria = {
+            "startTimeFromAscendingNode": start,
+            "completionTimeFromAscendingNode": end,
+            "geom": geom,
+        }
+        self.eodag.set_preferred_provider(provider)
+        results = self.eodag.search_all(
+            productType=product_type, items_per_page=items_per_page, **search_criteria
+        )
+        if check_products:
+            self.assertGreater(len(results), 0)
+            one_product = results[0]
+            self.assertEqual(one_product.provider, provider)
+        return results
+
 
 # @unittest.skip("skip auto run")
 class TestEODagEndToEnd(EndToEndBase):
@@ -357,6 +389,32 @@ class TestEODagEndToEnd(EndToEndBase):
         )
         self.assertGreaterEqual(os.stat(quicklook_file_path).st_size, 2 ** 5)
 
+    def test__search_by_id_sobloo(self):
+        # A single test with sobloo to check that _search_by_id returns
+        # correctly the exact product looked for.
+        uid = "S2A_MSIL1C_20200810T030551_N0209_R075_T53WPU_20200810T050611"
+        provider = "sobloo"
+
+        products, _ = self.eodag._search_by_id(uid=uid, provider=provider)
+        product = products[0]
+
+        self.assertEqual(product.properties["id"], uid)
+
+    def test_end_to_end_search_all_mundi_default(self):
+        # 23/03/2021: Got 16 products for this search
+        results = self.execute_search_all(*MUNDI_SEARCH_ARGS)
+        self.assertGreater(len(results), 10)
+
+    def test_end_to_end_search_all_mundi_iterate(self):
+        # 23/03/2021: Got 16 products for this search
+        results = self.execute_search_all(*MUNDI_SEARCH_ARGS, items_per_page=10)
+        self.assertGreater(len(results), 10)
+
+    def test_end_to_end_search_all_astraea_eod_iterate(self):
+        # 23/03/2021: Got 39 products for this search
+        results = self.execute_search_all(*ASTRAE_EOD_SEARCH_ARGS, items_per_page=10)
+        self.assertGreater(len(results), 10)
+
 
 class TestEODagEndToEndWrongCredentials(EndToEndBase):
     """Make real case tests with wrong credentials. This assumes the existence of a
diff --git a/tests/units/test_core.py b/tests/units/test_core.py
index df78fce7..221f70bc 100644
--- a/tests/units/test_core.py
+++ b/tests/units/test_core.py
@@ -17,17 +17,24 @@
 # limitations under the License.
 
 import glob
+import json
 import os
 import shutil
 import unittest
+from copy import deepcopy
 
 from shapely import wkt
-from shapely.geometry import LineString, MultiPolygon
+from shapely.geometry import LineString, MultiPolygon, Polygon
 
 from eodag.utils import GENERIC_PRODUCT_TYPE
 from tests import TEST_RESOURCES_PATH
 from tests.context import (
+    DEFAULT_MAX_ITEMS_PER_PAGE,
     EODataAccessGateway,
+    EOProduct,
+    NoMatchingProductType,
+    PluginImplementationError,
+    SearchResult,
     UnsupportedProvider,
     get_geometry_from_various,
     makedirs,
@@ -143,25 +150,10 @@ class TestCore(unittest.TestCase):
         "earth_search",
     ]
 
-    def setUp(self):
-        super(TestCore, self).setUp()
-        self.dag = EODataAccessGateway()
-
-    def tearDown(self):
-        super(TestCore, self).tearDown()
-        for old in glob.glob1(self.dag.conf_dir, "*.old") + glob.glob1(
-            self.dag.conf_dir, ".*.old"
-        ):
-            old_path = os.path.join(self.dag.conf_dir, old)
-            if os.path.exists(old_path):
-                try:
-                    os.remove(old_path)
-                except OSError:
-                    shutil.rmtree(old_path)
-        if os.getenv("EODAG_CFG_FILE") is not None:
-            os.environ.pop("EODAG_CFG_FILE")
-        if os.getenv("EODAG_LOCS_CFG_FILE") is not None:
-            os.environ.pop("EODAG_LOCS_CFG_FILE")
+    @classmethod
+    def setUpClass(cls):
+        cls.dag = EODataAccessGateway()
+        cls.conf_dir = os.path.join(os.path.expanduser("~"), ".config", "eodag")
 
     def test_supported_providers_in_unit_test(self):
         """Every provider must be referenced in the core unittest SUPPORTED_PROVIDERS class attribute"""  # noqa
@@ -216,25 +208,75 @@ class TestCore(unittest.TestCase):
         self.assertIn("sensorType", structure)
         self.assertIn(structure["ID"], self.SUPPORTED_PRODUCT_TYPES)
 
-    def test_core_object_creates_config_standard_location(self):
-        """The core object must create a user config file in standard user config location on instantiation"""  # noqa
-        self.execution_involving_conf_dir(inspect="eodag.yml")
-
-    def test_core_object_creates_index_if_not_exist(self):
-        """The core object must create an index in user config directory"""
-        self.execution_involving_conf_dir(inspect=".index")
-
     @mock.patch("eodag.api.core.open_dir", autospec=True)
     @mock.patch("eodag.api.core.exists_in", autospec=True, return_value=True)
     def test_core_object_open_index_if_exists(self, exists_in_mock, open_dir_mock):
         """The core object must use the existing index dir if any"""
-        conf_dir = os.path.join(os.path.expanduser("~"), ".config", "eodag")
-        index_dir = os.path.join(conf_dir, ".index")
+        index_dir = os.path.join(self.conf_dir, ".index")
         if not os.path.exists(index_dir):
             makedirs(index_dir)
         EODataAccessGateway()
         open_dir_mock.assert_called_with(index_dir)
 
+    def test_core_object_set_default_locations_config(self):
+        """The core object must set the default locations config on instantiation"""  # noqa
+        default_shpfile = os.path.join(
+            self.conf_dir, "shp", "ne_110m_admin_0_map_units.shp"
+        )
+        self.assertIsInstance(self.dag.locations_config, list)
+        self.assertEqual(
+            self.dag.locations_config,
+            [dict(attr="ADM0_A3_US", name="country", path=default_shpfile)],
+        )
+
+    def test_core_object_locations_file_not_found(self):
+        """The core object must set the locations to an empty list when the file is not found"""  # noqa
+        dag = EODataAccessGateway(locations_conf_path="no_locations.yml")
+        self.assertEqual(dag.locations_config, [])
+
+    def test_rebuild_index(self):
+        """Change eodag version and check that whoosh index is rebuilt"""
+        index_dir = os.path.join(self.dag.conf_dir, ".index")
+        index_dir_mtime = os.path.getmtime(index_dir)
+
+        self.assertNotEqual(self.dag.get_version(), "fake-version")
+
+        with mock.patch(
+            "eodag.api.core.EODataAccessGateway.get_version",
+            autospec=True,
+            return_value="fake-version",
+        ):
+            self.assertEqual(self.dag.get_version(), "fake-version")
+
+            self.dag.build_index()
+
+            # check that index_dir has beeh re-created
+            self.assertNotEqual(os.path.getmtime(index_dir), index_dir_mtime)
+
+
+class TestCoreConfWithEnvVar(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.dag = EODataAccessGateway()
+
+    @classmethod
+    def tearDownClass(cls):
+        if os.getenv("EODAG_CFG_FILE") is not None:
+            os.environ.pop("EODAG_CFG_FILE")
+        if os.getenv("EODAG_LOCS_CFG_FILE") is not None:
+            os.environ.pop("EODAG_LOCS_CFG_FILE")
+
+    def test_core_object_prioritize_locations_file_in_envvar(self):
+        """The core object must use the locations file pointed to by the EODAG_LOCS_CFG_FILE env var"""  # noqa
+        os.environ["EODAG_LOCS_CFG_FILE"] = os.path.join(
+            TEST_RESOURCES_PATH, "file_locations_override.yml"
+        )
+        dag = EODataAccessGateway()
+        self.assertEqual(
+            dag.locations_config,
+            [dict(attr="dummyattr", name="dummyname", path="dummypath.shp")],
+        )
+
     def test_core_object_prioritize_config_file_in_envvar(self):
         """The core object must use the config file pointed to by the EODAG_CFG_FILE env var"""  # noqa
         os.environ["EODAG_CFG_FILE"] = os.path.join(
@@ -246,6 +288,24 @@ class TestCore(unittest.TestCase):
         # peps outputs prefix is set to /data
         self.assertEqual(dag.providers_config["peps"].download.outputs_prefix, "/data")
 
+
+class TestCoreInvolvingConfDir(unittest.TestCase):
+    def setUp(self):
+        super(TestCoreInvolvingConfDir, self).setUp()
+        self.dag = EODataAccessGateway()
+
+    def tearDown(self):
+        super(TestCoreInvolvingConfDir, self).tearDown()
+        for old in glob.glob1(self.dag.conf_dir, "*.old") + glob.glob1(
+            self.dag.conf_dir, ".*.old"
+        ):
+            old_path = os.path.join(self.dag.conf_dir, old)
+            if os.path.exists(old_path):
+                try:
+                    os.remove(old_path)
+                except OSError:
+                    shutil.rmtree(old_path)
+
     def execution_involving_conf_dir(self, inspect=None):
         """Check that the path(s) inspected (str, list) are created after the instantation
         of EODataAccessGateway. If they were already there, rename them (.old), instantiate,
@@ -273,36 +333,23 @@ class TestCore(unittest.TestCase):
                         os.unlink(current)
                     shutil.move(old, current)
 
+    def test_core_object_creates_config_standard_location(self):
+        """The core object must create a user config file in standard user config location on instantiation"""  # noqa
+        self.execution_involving_conf_dir(inspect="eodag.yml")
+
+    def test_core_object_creates_index_if_not_exist(self):
+        """The core object must create an index in user config directory"""
+        self.execution_involving_conf_dir(inspect=".index")
+
     def test_core_object_creates_locations_standard_location(self):
         """The core object must create a locations config file and a shp dir in standard user config location on instantiation"""  # noqa
         self.execution_involving_conf_dir(inspect=["locations.yml", "shp"])
 
-    def test_core_object_set_default_locations_config(self):
-        """The core object must set the default locations config on instantiation"""  # noqa
-        dag = EODataAccessGateway()
-        conf_dir = os.path.join(os.path.expanduser("~"), ".config", "eodag")
-        default_shpfile = os.path.join(conf_dir, "shp", "ne_110m_admin_0_map_units.shp")
-        self.assertIsInstance(dag.locations_config, list)
-        self.assertEqual(
-            dag.locations_config,
-            [dict(attr="ADM0_A3_US", name="country", path=default_shpfile)],
-        )
-
-    def test_core_object_locations_file_not_found(self):
-        """The core object must set the locations to an empty list when the file is not found"""  # noqa
-        dag = EODataAccessGateway(locations_conf_path="no_locations.yml")
-        self.assertEqual(dag.locations_config, [])
 
-    def test_core_object_prioritize_locations_file_in_envvar(self):
-        """The core object must use the locations file pointed to by the EODAG_LOCS_CFG_FILE env var"""  # noqa
-        os.environ["EODAG_LOCS_CFG_FILE"] = os.path.join(
-            TEST_RESOURCES_PATH, "file_locations_override.yml"
-        )
-        dag = EODataAccessGateway()
-        self.assertEqual(
-            dag.locations_config,
-            [dict(attr="dummyattr", name="dummyname", path="dummypath.shp")],
-        )
+class TestCoreGeometry(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.dag = EODataAccessGateway()
 
     def test_get_geometry_from_various_no_locations(self):
         """The search geometry can be set from a dict, list, tuple, WKT string or shapely geom"""
@@ -415,9 +462,8 @@ class TestCore(unittest.TestCase):
             locations_config, locations=dict(country="FRA"), geometry=geometry
         )
         self.assertIsInstance(geom_combined, MultiPolygon)
-        self.assertEquals(
-            len(geom_combined), 4
-        )  # France + Guyana + Corsica + somewhere over Poland
+        # France + Guyana + Corsica + somewhere over Poland
+        self.assertEquals(len(geom_combined), 4)
         geometry = {
             "lonmin": 0,
             "latmin": 50,
@@ -428,26 +474,591 @@ class TestCore(unittest.TestCase):
             locations_config, locations=dict(country="FRA"), geometry=geometry
         )
         self.assertIsInstance(geom_combined, MultiPolygon)
-        self.assertEquals(
-            len(geom_combined), 3
-        )  # The bounding box overlaps with France inland
+        # The bounding box overlaps with France inland
+        self.assertEquals(len(geom_combined), 3)
 
-    def test_rebuild_index(self):
-        """Change eodag version and check that whoosh index is rebuilt"""
 
-        index_dir = os.path.join(self.dag.conf_dir, ".index")
-        index_dir_mtime = os.path.getmtime(index_dir)
+class TestCoreSearch(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.dag = EODataAccessGateway()
+        # Get a SearchResult obj with 2 S2_MSI_L1C peps products
+        search_results_file = os.path.join(
+            TEST_RESOURCES_PATH, "eodag_search_result_peps.geojson"
+        )
+        with open(search_results_file, encoding="utf-8") as f:
+            search_results_geojson = json.load(f)
+        cls.search_results = SearchResult.from_geojson(search_results_geojson)
+        cls.search_results_size = len(cls.search_results)
+        # Change the id of these products, to emulate different products
+        search_results_data_2 = deepcopy(cls.search_results.data)
+        search_results_data_2[0].properties["id"] = "a"
+        search_results_data_2[1].properties["id"] = "b"
+        cls.search_results_2 = SearchResult(search_results_data_2)
+        cls.search_results_size_2 = len(cls.search_results_2)
 
-        self.assertNotEqual(self.dag.get_version(), "fake-version")
+    def test_guess_product_type_with_kwargs(self):
+        """guess_product_type must return the products matching the given kwargs"""
+        kwargs = dict(
+            instrument="MSI",
+            platform="SENTINEL2",
+            platformSerialIdentifier="S2A",
+        )
+        actual = self.dag.guess_product_type(**kwargs)
+        expected = [
+            "S2_MSI_L1C",
+            "S2_MSI_L2A",
+            "S2_MSI_L2A_MAJA",
+            "S2_MSI_L2B_MAJA_SNOW",
+            "S2_MSI_L2B_MAJA_WATER",
+            "S2_MSI_L3A_WASP",
+        ]
+        self.assertEqual(actual, expected)
 
-        with mock.patch(
-            "eodag.api.core.EODataAccessGateway.get_version",
-            autospec=True,
-            return_value="fake-version",
-        ):
-            self.assertEqual(self.dag.get_version(), "fake-version")
+    def test_guess_product_type_without_kwargs(self):
+        """guess_product_type must raise an exception when no kwargs are provided"""
+        with self.assertRaises(NoMatchingProductType):
+            self.dag.guess_product_type()
 
-            self.dag.build_index()
+    def test__prepare_search_no_parameters(self):
+        """_prepare_search must create some kwargs even when no parameter has been provided"""  # noqa
+        prepared_search = self.dag._prepare_search()
+        expected = {
+            "geometry": None,
+            "productType": None,
+        }
+        self.assertDictContainsSubset(expected, prepared_search)
+        expected = set(["geometry", "productType", "auth", "search_plugin"])
+        self.assertSetEqual(expected, set(prepared_search))
 
-            # check that index_dir has beeh re-created
-            self.assertNotEqual(os.path.getmtime(index_dir), index_dir_mtime)
+    def test__prepare_search_dates(self):
+        """_prepare_search must handle start & end dates"""
+        base = {
+            "start": "2020-01-01",
+            "end": "2020-02-01",
+        }
+        prepared_search = self.dag._prepare_search(**base)
+        expected = {
+            "startTimeFromAscendingNode": base["start"],
+            "completionTimeFromAscendingNode": base["end"],
+        }
+        self.assertDictContainsSubset(expected, prepared_search)
+
+    def test__prepare_search_geom(self):
+        """_prepare_search must handle geom, box and bbox"""
+        # The default way to provide a geom is through the 'geom' argument.
+        base = {"geom": (0, 50, 2, 52)}
+        # "geom": "POLYGON ((1 43, 1 44, 2 44, 2 43, 1 43))"
+        prepared_search = self.dag._prepare_search(**base)
+        self.assertIn("geometry", prepared_search)
+        # 'box' and 'bbox' are supported for backwards compatibility,
+        # The priority is geom > bbox > box
+        base = {"box": (0, 50, 2, 52)}
+        prepared_search = self.dag._prepare_search(**base)
+        self.assertIn("geometry", prepared_search)
+        base = {"bbox": (0, 50, 2, 52)}
+        prepared_search = self.dag._prepare_search(**base)
+        self.assertIn("geometry", prepared_search)
+        base = {
+            "geom": "POLYGON ((1 43, 1 44, 2 44, 2 43, 1 43))",
+            "bbox": (0, 50, 2, 52),
+            "box": (0, 50, 1, 51),
+        }
+        prepared_search = self.dag._prepare_search(**base)
+        self.assertIn("geometry", prepared_search)
+        self.assertNotIn("bbox", prepared_search)
+        self.assertNotIn("bbox", prepared_search)
+        self.assertIsInstance(prepared_search["geometry"], Polygon)
+
+    def test__prepare_search_locations(self):
+        """_prepare_search must handle a location search"""
+        # When locations where introduced they could be passed
+        # as regular kwargs. The new and recommended way to provide
+        # them is through the 'locations' parameter.
+        base = {"locations": {"country": "FRA"}}
+        prepared_search = self.dag._prepare_search(**base)
+        self.assertIn("geometry", prepared_search)
+
+        # TODO: Remove this when support for locations kwarg is dropped.
+        base = {"country": "FRA"}
+        prepared_search = self.dag._prepare_search(**base)
+        self.assertIn("geometry", prepared_search)
+        self.assertNotIn("country", prepared_search)
+
+    def test__prepare_search_product_type_provided(self):
+        """_prepare_search must handle when a product type is given"""
+        base = {"productType": "S2_MSI_L1C"}
+        prepared_search = self.dag._prepare_search(**base)
+        expected = {"productType": base["productType"]}
+        self.assertDictContainsSubset(expected, prepared_search)
+
+    def test__prepare_search_product_type_guess_it(self):
+        """_prepare_search must guess a product type when required to"""
+        # Uses guess_product_type to find the product matching
+        # the best the given params.
+        base = dict(
+            instrument="MSI",
+            platform="SENTINEL2",
+            platformSerialIdentifier="S2A",
+        )
+        prepared_search = self.dag._prepare_search(**base)
+        expected = {"productType": "S2_MSI_L1C", **base}
+        self.assertDictContainsSubset(expected, prepared_search)
+
+    def test__prepare_search_with_id(self):
+        """_prepare_search must handle a search by id"""
+        base = {"id": "dummy-id", "provider": "sobloo"}
+        prepared_search = self.dag._prepare_search(**base)
+        expected = base
+        self.assertDictEqual(expected, prepared_search)
+
+    def test__prepare_search_preserve_additional_kwargs(self):
+        """_prepare_search must preserve additional kwargs"""
+        base = {
+            "productType": "S2_MSI_L1C",
+            "cloudCover": 10,
+        }
+        prepared_search = self.dag._prepare_search(**base)
+        expected = base
+        self.assertDictContainsSubset(expected, prepared_search)
+
+    def test__prepare_search_search_plugin_has_known_product_properties(self):
+        """_prepare_search must attach the product properties to the search plugin"""
+        prev_fav_provider = self.dag.get_preferred_provider()[0]
+        try:
+            self.dag.set_preferred_provider("peps")
+            base = {"productType": "S2_MSI_L1C"}
+            prepared_search = self.dag._prepare_search(**base)
+            # Just check that the title has been set correctly. There are more (e.g.
+            # abstract, platform, etc.) but this is sufficient to check that the
+            # product_type_config dict has been created and populated.
+            self.assertEqual(
+                prepared_search["search_plugin"].config.product_type_config["title"],
+                "SENTINEL2 Level-1C",
+            )
+        finally:
+            self.dag.set_preferred_provider(prev_fav_provider)
+
+    def test__prepare_search_search_plugin_has_generic_product_properties(self):
+        """_prepare_search must be able to attach the generic product properties to the search plugin"""  # noqa
+        prev_fav_provider = self.dag.get_preferred_provider()[0]
+        try:
+            self.dag.set_preferred_provider("peps")
+            base = {"productType": "product_unknown_to_eodag"}
+            prepared_search = self.dag._prepare_search(**base)
+            # product_type_config is still created if the product is not known to eodag
+            # however it contains no data.
+            self.assertIsNone(
+                prepared_search["search_plugin"].config.product_type_config["title"],
+            )
+        finally:
+            self.dag.set_preferred_provider(prev_fav_provider)
+
+    def test__prepare_search_peps_plugins_product_available(self):
+        """_prepare_search must return the search and auth plugins when productType is defined"""  # noqa
+        prev_fav_provider = self.dag.get_preferred_provider()[0]
+        try:
+            self.dag.set_preferred_provider("peps")
+            base = {"productType": "S2_MSI_L1C"}
+            prepared_search = self.dag._prepare_search(**base)
+            self.assertEqual(prepared_search["search_plugin"].provider, "peps")
+            self.assertEqual(prepared_search["auth"].provider, "peps")
+        finally:
+            self.dag.set_preferred_provider(prev_fav_provider)
+
+    def test__prepare_search_no_plugins_when_search_by_id(self):
+        """_prepare_search must not return the search and auth plugins for a search by id"""  # noqa
+        base = {"id": "some_id", "provider": "some_provider"}
+        prepared_search = self.dag._prepare_search(**base)
+        self.assertNotIn("search_plugin", prepared_search)
+        self.assertNotIn("auth", prepared_search)
+
+    def test__prepare_search_peps_plugins_product_not_available(self):
+        """_prepare_search can use another set of search and auth plugins than the ones of the preferred provider"""  # noqa
+        # Document a special behaviour whereby the search and auth plugins don't
+        # correspond to the preferred one. This occurs whenever the searched product
+        # isn't available for the preferred provider but is made available by  another
+        # one. In that case peps provides it and happens to be the first one on the list
+        # of providers that make it available.
+        prev_fav_provider = self.dag.get_preferred_provider()[0]
+        try:
+            self.dag.set_preferred_provider("theia")
+            base = {"productType": "S2_MSI_L1C"}
+            prepared_search = self.dag._prepare_search(**base)
+            self.assertEqual(prepared_search["search_plugin"].provider, "peps")
+            self.assertEqual(prepared_search["auth"].provider, "peps")
+        finally:
+            self.dag.set_preferred_provider(prev_fav_provider)
+
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test__do_search_counts_by_default(self, search_plugin):
+        """__do_search must create a count query by default"""
+        search_plugin.provider = "peps"
+        search_plugin.query.return_value = (
+            self.search_results.data,  # a list must be returned by .query
+            self.search_results_size,
+        )
+        sr, estimate = self.dag._do_search(search_plugin=search_plugin)
+        self.assertIsInstance(sr, SearchResult)
+        self.assertEqual(len(sr), self.search_results_size)
+        self.assertEqual(estimate, self.search_results_size)
+
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test__do_search_without_count(self, search_plugin):
+        """__do_search must be able to create a query without a count"""
+        search_plugin.provider = "peps"
+        search_plugin.query.return_value = (
+            self.search_results.data,
+            None,  # .query must return None if count is False
+        )
+        sr, estimate = self.dag._do_search(search_plugin=search_plugin, count=False)
+        self.assertIsNone(estimate)
+        self.assertEqual(len(sr), self.search_results_size)
+
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test__do_search_paginated_handle_no_count_returned(self, search_plugin):
+        """__do_search must provide a best estimate when a provider doesn't return a count"""  # noqa
+        search_plugin.provider = "peps"
+        # If the provider doesn't return a count, .query returns 0
+        search_plugin.query.return_value = (self.search_results.data, 0)
+        page = 4
+        sr, estimate = self.dag._do_search(
+            search_plugin=search_plugin,
+            page=page,
+            items_per_page=2,
+        )
+        self.assertEqual(len(sr), self.search_results_size)
+        # The count guess is: page * number_of_products_returned
+        self.assertEqual(estimate, page * self.search_results_size)
+
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test__do_search_paginated_handle_fuzzy_count(self, search_plugin):
+        """__do_search must provide a best estimate when a provider returns a fuzzy count"""  # noqa
+        search_plugin.provider = "peps"
+        search_plugin.query.return_value = (
+            self.search_results.data * 4,  # 8 products returned
+            22,  # fuzzy number, less than the real total count
+        )
+        page = 4
+        items_per_page = 10
+        sr, estimate = self.dag._do_search(
+            search_plugin=search_plugin,
+            page=page,
+            items_per_page=items_per_page,
+        )
+        # At page 4 with 10 items_per_page we should have a count of at least 30
+        # products available. However the provider returned 22. We know it's wrong.
+        # So we update the count with our current knowledge: 30 + 8
+        # Note that this estimate could still be largely inferior to the real total
+        # count.
+        expected_estimate = items_per_page * (page - 1) + len(sr)
+        self.assertEqual(len(sr), 8)
+        self.assertEqual(estimate, expected_estimate)
+
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test__do_search_paginated_handle_null_count(self, search_plugin):
+        """__do_search must provide a best estimate when a provider returns a null count"""  # noqa
+        # TODO: check the underlying implementation, it doesn't make so much sense since
+        # this case is already covered with nb_res = len(res) * page. This one uses
+        # nb_res = items_per_page * (page - 1) whick actually makes more sense. Choose
+        # one of them.
+        search_plugin.provider = "peps"
+        search_plugin.query.return_value = ([], 0)
+        page = 4
+        items_per_page = 10
+        sr, estimate = self.dag._do_search(
+            search_plugin=search_plugin,
+            page=page,
+            items_per_page=items_per_page,
+        )
+        expected_estimate = items_per_page * (page - 1)
+        self.assertEqual(len(sr), 0)
+        self.assertEqual(estimate, expected_estimate)
+
+    def test__do_search_does_not_raise_by_default(self):
+        """__do_search must not raise any error by default"""
+        # provider attribute required internally by __do_search for logging purposes.
+        class DummySearchPlugin:
+            provider = "peps"
+
+        sr, estimate = self.dag._do_search(search_plugin=DummySearchPlugin())
+        self.assertIsInstance(sr, SearchResult)
+        self.assertEqual(len(sr), 0)
+        self.assertEqual(estimate, 0)
+
+    def test__do_search_can_raise_errors(self):
+        """__do_search must not raise any error by default"""
+
+        class DummySearchPlugin:
+            provider = "peps"
+
+        # AttributeError raised when .query is tried to be accessed on the dummy plugin.
+        with self.assertRaises(AttributeError):
+            self.dag._do_search(search_plugin=DummySearchPlugin(), raise_errors=True)
+
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test__do_search_query_products_must_be_a_list(self, search_plugin):
+        """__do_search expects that each search plugin returns a list of products."""
+        search_plugin.provider = "peps"
+        search_plugin.query.return_value = (
+            self.search_results,
+            self.search_results_size,
+        )
+        with self.assertRaises(PluginImplementationError):
+            self.dag._do_search(search_plugin=search_plugin, raise_errors=True)
+
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test__do_search_register_downloader_if_search_intersection(self, search_plugin):
+        """__do_search must register each product's downloader if search_intersection is not None"""  # noqa
+        search_plugin.provider = "peps"
+        search_plugin.query.return_value = (
+            self.search_results.data,
+            self.search_results_size,
+        )
+        sr, _ = self.dag._do_search(search_plugin=search_plugin)
+        for product in sr:
+            self.assertIsNotNone(product.downloader)
+
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test__do_search_doest_not_register_downloader_if_no_search_intersection(
+        self, search_plugin
+    ):
+        """__do_search must not register downloaders if search_intersection is None"""
+
+        class DummyProduct:
+            seach_intersecion = None
+
+        search_plugin.provider = "peps"
+        search_plugin.query.return_value = ([DummyProduct(), DummyProduct()], 2)
+        sr, _ = self.dag._do_search(search_plugin=search_plugin)
+        for product in sr:
+            self.assertIsNone(product.downloader)
+
+    @mock.patch("eodag.api.core.EODataAccessGateway._prepare_search", autospec=True)
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test_search_iter_page_returns_iterator(self, search_plugin, prepare_seach):
+        """search_iter_page must return an iterator"""
+        search_plugin.provider = "peps"
+        search_plugin.query.side_effect = [
+            (self.search_results.data, None),
+            (self.search_results_2.data, None),
+        ]
+
+        class DummyConfig:
+            pagination = {}
+
+        search_plugin.config = DummyConfig()
+        prepare_seach.return_value = dict(search_plugin=search_plugin)
+        page_iterator = self.dag.search_iter_page(items_per_page=2)
+        first_result_page = next(page_iterator)
+        self.assertIsInstance(first_result_page, SearchResult)
+        self.assertEqual(len(first_result_page), self.search_results_size)
+        second_result_page = next(page_iterator)
+        self.assertIsInstance(second_result_page, SearchResult)
+        self.assertEqual(len(second_result_page), self.search_results_size_2)
+
+    @mock.patch("eodag.api.core.EODataAccessGateway._prepare_search", autospec=True)
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test_search_iter_page_exhaust_get_all_pages_and_quit_early(
+        self, search_plugin, prepare_seach
+    ):
+        """search_iter_page must stop as soon as less than items_per_page products were retrieved"""  # noqa
+        search_plugin.provider = "peps"
+        search_plugin.query.side_effect = [
+            (self.search_results.data, None),
+            ([self.search_results_2.data[0]], None),
+        ]
+
+        class DummyConfig:
+            pagination = {}
+
+        search_plugin.config = DummyConfig()
+        prepare_seach.return_value = dict(search_plugin=search_plugin)
+        page_iterator = self.dag.search_iter_page(items_per_page=2)
+        all_page_results = list(page_iterator)
+        self.assertEqual(len(all_page_results), 2)
+        self.assertIsInstance(all_page_results[0], SearchResult)
+
+    @mock.patch("eodag.api.core.EODataAccessGateway._prepare_search", autospec=True)
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test_search_iter_page_exhaust_get_all_pages_no_products_last_page(
+        self, search_plugin, prepare_seach
+    ):
+        """search_iter_page must stop if the page doesn't return any product"""
+        search_plugin.provider = "peps"
+        search_plugin.query.side_effect = [
+            (self.search_results.data, None),
+            (self.search_results_2.data, None),
+            ([], None),
+        ]
+
+        class DummyConfig:
+            pagination = {}
+
+        search_plugin.config = DummyConfig()
+        prepare_seach.return_value = dict(search_plugin=search_plugin)
+        page_iterator = self.dag.search_iter_page(items_per_page=2)
+        all_page_results = list(page_iterator)
+        self.assertEqual(len(all_page_results), 2)
+        self.assertIsInstance(all_page_results[0], SearchResult)
+
+    @mock.patch("eodag.api.core.EODataAccessGateway._prepare_search", autospec=True)
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test_search_iter_page_does_not_handle_query_errors(
+        self, search_plugin, prepare_seach
+    ):
+        """search_iter_page must propagate errors"""
+        search_plugin.provider = "peps"
+        search_plugin.query.side_effect = AttributeError()
+        prepare_seach.return_value = dict(search_plugin=search_plugin)
+        page_iterator = self.dag.search_iter_page()
+        with self.assertRaises(AttributeError):
+            next(page_iterator)
+
+    @mock.patch(
+        "eodag.plugins.search.qssearch.QueryStringSearch._request", autospec=True
+    )
+    @mock.patch(
+        "eodag.plugins.search.qssearch.QueryStringSearch.normalize_results",
+        autospec=True,
+    )
+    def test_search_iter_page_must_reset_next_attrs_if_next_mechanism(
+        self, normalize_results, _request
+    ):
+        """search_iter_page must reset the search plugin if the next mechanism is used"""
+        # More specifically: next_page_url must be None and
+        # config.pagination["next_page_url_tpl"] must be equal to its original value.
+        _request.return_value.json.return_value = {
+            "features": [],
+            "links": [{"rel": "next", "href": "url/to/next/page"}],
+        }
+
+        p1 = EOProduct("dummy", dict(geometry="POINT (0 0)", id="1"))
+        p1.search_intersection = None
+        p2 = EOProduct("dummy", dict(geometry="POINT (0 0)", id="2"))
+        p2.search_intersection = None
+        normalize_results.side_effect = [[p1], [p2]]
+        dag = EODataAccessGateway()
+        dummy_provider_config = """
+        dummy_provider:
+            search:
+                type: QueryStringSearch
+                api_endpoint: https://api.my_new_provider/search
+                pagination:
+                    next_page_url_tpl: 'dummy_next_page_url_tpl'
+                    next_page_url_key_path: '$.links[?(@.rel="next")].href'
+                metadata_mapping:
+                    dummy: 'dummy'
+            products:
+                S2_MSI_L1C:
+                    productType: '{productType}'
+        """
+        dag.update_providers_config(dummy_provider_config)
+        dag.set_preferred_provider("dummy_provider")
+
+        page_iterator = dag.search_iter_page(productType="S2_MSI_L1C")
+        next(page_iterator)
+        search_plugin = next(
+            dag._plugins_manager.get_search_plugins(product_type="S2_MSI_L1C")
+        )
+        self.assertIsNone(search_plugin.next_page_url)
+        self.assertEqual(
+            search_plugin.config.pagination["next_page_url_tpl"],
+            "dummy_next_page_url_tpl",
+        )
+
+    @mock.patch("eodag.api.core.EODataAccessGateway._prepare_search", autospec=True)
+    @mock.patch("eodag.plugins.search.qssearch.QueryStringSearch", autospec=True)
+    def test_search_all_must_collect_them_all(self, search_plugin, prepare_seach):
+        """search_all must return all the products available"""  # noqa
+        search_plugin.provider = "peps"
+        search_plugin.query.side_effect = [
+            (self.search_results.data, None),
+            ([self.search_results_2.data[0]], None),
+        ]
+
+        class DummyConfig:
+            pagination = {}
+
+        search_plugin.config = DummyConfig()
+
+        # Infinite generator her because passing directly the dict to
+        # prepare_search.return_value (or side_effect) didn't work. One function
+        # would do a dict.pop("search_plugin") that would remove the item from the
+        # mocked return value. Later calls would then break
+        def yield_search_plugin():
+            while True:
+                yield {"search_plugin": search_plugin}
+
+        prepare_seach.side_effect = yield_search_plugin()
+        all_results = self.dag.search_all(items_per_page=2)
+        self.assertIsInstance(all_results, SearchResult)
+        self.assertEqual(len(all_results), 3)
+
+    @mock.patch("eodag.api.core.EODataAccessGateway.search_iter_page", autospec=True)
+    def test_search_all_use_max_items_per_page(self, mocked_search_iter_page):
+        """search_all must use the configured parameter max_items_per_page if available"""  # noqa
+        dag = EODataAccessGateway()
+        dummy_provider_config = """
+        dummy_provider:
+            search:
+                type: QueryStringSearch
+                api_endpoint: https://api.my_new_provider/search
+                pagination:
+                    max_items_per_page: 2
+                metadata_mapping:
+                    dummy: 'dummy'
+            products:
+                S2_MSI_L1C:
+                    productType: '{productType}'
+        """
+        mocked_search_iter_page.return_value = (self.search_results for _ in range(1))
+        dag.update_providers_config(dummy_provider_config)
+        dag.set_preferred_provider("dummy_provider")
+        dag.search_all(productType="S2_MSI_L1C")
+        self.assertEqual(mocked_search_iter_page.call_args[1]["items_per_page"], 2)
+
+    @mock.patch("eodag.api.core.EODataAccessGateway.search_iter_page", autospec=True)
+    def test_search_all_use_default_value(self, mocked_search_iter_page):
+        """search_all must use the DEFAULT_MAX_ITEMS_PER_PAGE if the provider's one wasn't configured"""  # noqa
+        dag = EODataAccessGateway()
+        dummy_provider_config = """
+        dummy_provider:
+            search:
+                type: QueryStringSearch
+                api_endpoint: https://api.my_new_provider/search
+                metadata_mapping:
+                    dummy: 'dummy'
+            products:
+                S2_MSI_L1C:
+                    productType: '{productType}'
+        """
+        mocked_search_iter_page.return_value = (self.search_results for _ in range(1))
+        dag.update_providers_config(dummy_provider_config)
+        dag.set_preferred_provider("dummy_provider")
+        dag.search_all(productType="S2_MSI_L1C")
+        self.assertEqual(
+            mocked_search_iter_page.call_args[1]["items_per_page"],
+            DEFAULT_MAX_ITEMS_PER_PAGE,
+        )
+
+    @mock.patch("eodag.api.core.EODataAccessGateway.search_iter_page", autospec=True)
+    def test_search_all_user_items_per_page(self, mocked_search_iter_page):
+        """search_all must use the value of items_per_page provided by the user"""
+        dag = EODataAccessGateway()
+        dummy_provider_config = """
+        dummy_provider:
+            search:
+                type: QueryStringSearch
+                api_endpoint: https://api.my_new_provider/search
+                metadata_mapping:
+                    dummy: 'dummy'
+            products:
+                S2_MSI_L1C:
+                    productType: '{productType}'
+        """
+        mocked_search_iter_page.return_value = (self.search_results for _ in range(1))
+        dag.update_providers_config(dummy_provider_config)
+        dag.set_preferred_provider("dummy_provider")
+        dag.search_all(productType="S2_MSI_L1C", items_per_page=7)
+        self.assertEqual(mocked_search_iter_page.call_args[1]["items_per_page"], 7)