Repository: CS-SI/eodag (python)

Issue:
Add to the search command the ability to search for all the products matching some criteria
The API will soon get a `search_all` method that will allow to collect at once all the products from a provider that match a set of criteria. The CLI should also benefit from this enhancement. I suggest that we add a `--all` option to the `search` command. Internally instead of calling `search` it would call `search_all`.

Pull request:
Add a --all option to the search command to search for all the products
Fixes #202 

I had to remove the default value of `--items` at the "click" level, since it's not used in the same way by `search` and `search_all`. Instead it is set inside the CLI search function. I have adapted the "help" of `--items` so that it still displays the default value (as of now: 20).

Depends on #190 

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 59bcdfc1..9998e581 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -1,6 +1,11 @@
 Release history
 ---------------
 
+2.2.0 (2021-xx-xx)
+++++++++++++++++++
+
+- More explicit signature for ``setup_logging``, fixes `GH-197 <https://github.com/CS-SI/eodag/issues/197>`_
+
 2.1.1 (2021-03-18)
 ++++++++++++++++++
 
diff --git a/README.rst b/README.rst
index bf279ae5..6404fd97 100644
--- a/README.rst
+++ b/README.rst
@@ -178,22 +178,23 @@ Then you can start playing with it:
         --start 2018-01-01 \
         --end 2018-01-31 \
         --cloudCover 20 \
-        --productType S2_MSI_L1C
-        --cruncher FilterLatestIntersect \
+        --productType S2_MSI_L1C \
+        --all \
         --storage my_search.geojson
 
-The request above search for product types `S2_MSI_L1C` and will crunch the result using cruncher `FilterLatestIntersect`
-and storing the overall result to `my_search.geojson`.
+The request above searches for `S2_MSI_L1C` product types in a given bounding box, in January 2018. The command fetches internally all
+the products that match these criteria. Without `--all`, it would only fetch the products found on the first result page.
+It finally saves the results in a GeoJSON file.
 
 You can pass arguments to a cruncher on the command line by doing this (example with using `FilterOverlap` cruncher
 which takes `minimum_overlap` as argument)::
 
-        eodag search -f my_conf.yml -b 1 43 2 44 -s 2018-01-01 -e 2018-01-31 -p S2_MSI_L1C \
+        eodag search -f my_conf.yml -b 1 43 2 44 -s 2018-01-01 -e 2018-01-31 -p S2_MSI_L1C --all \
                      --cruncher FilterOverlap \
                      --cruncher-args FilterOverlap minimum_overlap 10
 
 The request above means : "Give me all the products of type `S2_MSI_L1C`, use `FilterOverlap` to keep only those products
-that are contained in the bbox I gave you, or whom spatial extent overlaps at least 10% (`minimum_overlap`) of the surface
+that are contained in the bbox I gave you, or whose spatial extent overlaps at least 10% (`minimum_overlap`) of the surface
 of this bbox"
 
 * To download the result of a previous call to `search`::
diff --git a/docs/use.rst b/docs/use.rst
index ef129fd0..42f0cb6a 100644
--- a/docs/use.rst
+++ b/docs/use.rst
@@ -172,23 +172,24 @@ Then you can start playing with it:
         --box 1 43 2 44 \
         --start 2018-01-01 --end 2018-01-31 \
         --productType S2_MSI_L1C \
-        --cruncher FilterLatestIntersect \
+        --all \
         --storage my_search.geojson
 
-The request above search for product types `S2_MSI_L1C` and will crunch the result using cruncher `FilterLatestIntersect`
-and storing the overall result to `my_search.geojson`.
+The request above searches for `S2_MSI_L1C` product types in a given bounding box, in January 2018. The command fetches internally all
+the products that match these criteria. Without `--all`, it would only fetch the products found on the first result page.
+It finally saves the results in a GeoJSON file.
 
 You can pass arguments to a cruncher on the command line by doing this (example with using `FilterOverlap` cruncher
 which takes `minimum_overlap` as argument):
 
 .. code-block:: console
 
-        eodag search -f my_conf.yml -b 1 43 2 44 -s 2018-01-01 -e 2018-01-31 -p S2_MSI_L1C \
+        eodag search -f my_conf.yml -b 1 43 2 44 -s 2018-01-01 -e 2018-01-31 -p S2_MSI_L1C --all \
                      --cruncher FilterOverlap \
                      --cruncher-args FilterOverlap minimum_overlap 10
 
 The request above means : "Give me all the products of type `S2_MSI_L1C`, use `FilterOverlap` to keep only those products
-that are contained in the bbox I gave you, or whom spatial extent overlaps at least 10% (`minimum_overlap`) of the surface
+that are contained in the bbox I gave you, or whose spatial extent overlaps at least 10% (`minimum_overlap`) of the surface
 of this bbox"
 
 You can use `eaodag search` with custom parameters. Custom parameters will be used as is in the query string search sent
diff --git a/eodag/api/core.py b/eodag/api/core.py
index 6b84203c..31a214cd 100644
--- a/eodag/api/core.py
+++ b/eodag/api/core.py
@@ -428,7 +428,7 @@ class EODataAccessGateway(object):
                     kwargs[search_key]
                 )
                 if results is None:
-                    results = searcher.search(query)
+                    results = searcher.search(query, limit=None)
                 else:
                     results.upgrade_and_extend(searcher.search(query))
             guesses = [r["ID"] for r in results or []]
diff --git a/eodag/cli.py b/eodag/cli.py
index b25bfa6b..9832e1c7 100755
--- a/eodag/cli.py
+++ b/eodag/cli.py
@@ -210,11 +210,12 @@ def version():
 @click.option(
     "--items",
     type=int,
-    default=DEFAULT_ITEMS_PER_PAGE,
-    show_default=True,
+    # default=DEFAULT_ITEMS_PER_PAGE,
+    show_default=False,
     help="The number of items to return. Eodag is bound to whatever limitation the "
     "providers have on the number of results they return. This option allows "
-    "to control how many items eodag should request",
+    "to control how many items eodag should request "
+    f"[default: {DEFAULT_ITEMS_PER_PAGE}]",
 )
 @click.option(
     "--page",
@@ -223,6 +224,15 @@ def version():
     show_default=True,
     help="Retrieve the given page",
 )
+@click.option(
+    "--all",
+    is_flag=True,
+    help="Retrieve ALL the products that match the search criteria. It collects "
+    "products by iterating over the results pages until no more products are available."
+    "At each iteration, the maximum number of items searched is either 'items' if set, "
+    "or a maximum value defined internally for the requested provider, or a default "
+    "maximum value equals to 50.",
+)
 @click.option(
     "--locations",
     type=str,
@@ -264,8 +274,7 @@ def search_crunch(ctx, **kwargs):
             click.echo(search_crunch.get_help(ctx))
         sys.exit(-1)
 
-    kwargs["verbose"] = ctx.obj["verbosity"]
-    setup_logging(**kwargs)
+    setup_logging(verbose=ctx.obj["verbosity"])
 
     if kwargs["box"] != (None,) * 4:
         rect = kwargs.pop("box")
@@ -333,10 +342,21 @@ def search_crunch(ctx, **kwargs):
     )
 
     # Search
-    results, total = gateway.search(
-        page=page, items_per_page=items_per_page, **criteria
-    )
-    click.echo("Found a total number of {} products".format(total))
+    get_all_products = kwargs.pop("all")
+    if get_all_products:
+        # search_all needs items_per_page to be None if the user lets eodag determines
+        # what value it should take.
+        items_per_page = None if items_per_page is None else items_per_page
+        results = gateway.search_all(items_per_page=items_per_page, **criteria)
+    else:
+        # search should better take a value that is not None
+        items_per_page = (
+            DEFAULT_ITEMS_PER_PAGE if items_per_page is None else items_per_page
+        )
+        results, total = gateway.search(
+            page=page, items_per_page=items_per_page, **criteria
+        )
+        click.echo("Found a total number of {} products".format(total))
     click.echo("Returned {} products".format(len(results)))
 
     # Crunch !
@@ -384,7 +404,7 @@ def list_pt(ctx, **kwargs):
             platformSerialIdentifier=kwargs.get("platformserialidentifier"),
             processingLevel=kwargs.get("processinglevel"),
             sensorType=kwargs.get("sensortype"),
-            **kwargs
+            **kwargs,
         )
     except NoMatchingProductType:
         if any(
@@ -457,8 +477,7 @@ def download(ctx, **kwargs):
             click.echo("Nothing to do (no search results file provided)")
             click.echo(download.get_help(ctx))
         sys.exit(1)
-    kwargs["verbose"] = ctx.obj["verbosity"]
-    setup_logging(**kwargs)
+    setup_logging(verbose=ctx.obj["verbosity"])
     conf_file = kwargs.pop("conf")
     if conf_file:
         conf_file = click.format_filename(conf_file)
diff --git a/eodag/utils/logging.py b/eodag/utils/logging.py
index 2a6ed5aa..12e629ea 100644
--- a/eodag/utils/logging.py
+++ b/eodag/utils/logging.py
@@ -19,15 +19,17 @@
 import logging.config
 
 
-def setup_logging(**kwargs):
+def setup_logging(verbose):
     """Define logging level
 
-    :param kwargs: Accepted Keyword arguments:
-                    verbose=0-3 logging level (None to max)
-    :type kwargs: dict
+    :param verbose: Accepted values:
+
+                    * 0: no logging
+                    * 1 or 2: INFO level
+                    * 3: DEBUG level
+    :type verbose: int
     """
-    verbosity = kwargs.pop("verbose")
-    if verbosity == 0:
+    if verbose == 0:
         logging.config.dictConfig(
             {
                 "version": 1,
@@ -40,7 +42,7 @@ def setup_logging(**kwargs):
                 },
             }
         )
-    elif verbosity <= 2:
+    elif verbose <= 2:
         logging.config.dictConfig(
             {
                 "version": 1,
@@ -71,7 +73,7 @@ def setup_logging(**kwargs):
                 },
             }
         )
-    elif verbosity == 3:
+    elif verbose == 3:
         logging.config.dictConfig(
             {
                 "version": 1,
@@ -102,3 +104,5 @@ def setup_logging(**kwargs):
                 },
             }
         )
+    else:
+        raise ValueError("'verbose' must be one of: 0, 1, 2, 3")


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index c3c7d742..94e07cc3 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -29,6 +29,7 @@ from pkg_resources import resource_filename
 from eodag.utils import GENERIC_PRODUCT_TYPE
 from tests import TEST_RESOURCES_PATH
 from tests.context import (
+    DEFAULT_ITEMS_PER_PAGE,
     AuthenticationError,
     MisconfiguredError,
     download,
@@ -153,7 +154,7 @@ class TestEodagCli(unittest.TestCase):
             )
             api_obj = dag.return_value
             api_obj.search.assert_called_once_with(
-                items_per_page=20,
+                items_per_page=DEFAULT_ITEMS_PER_PAGE,
                 page=1,
                 startTimeFromAscendingNode=None,
                 completionTimeFromAscendingNode=None,
@@ -198,7 +199,7 @@ class TestEodagCli(unittest.TestCase):
             )
             api_obj = dag.return_value
             api_obj.search.assert_called_once_with(
-                items_per_page=20,
+                items_per_page=DEFAULT_ITEMS_PER_PAGE,
                 page=1,
                 startTimeFromAscendingNode=None,
                 completionTimeFromAscendingNode=None,
@@ -295,7 +296,7 @@ class TestEodagCli(unittest.TestCase):
                 user_conf_file_path=conf_file, locations_conf_path=None
             )
             api_obj.search.assert_called_once_with(
-                items_per_page=20, page=1, **criteria
+                items_per_page=DEFAULT_ITEMS_PER_PAGE, page=1, **criteria
             )
             api_obj.crunch.assert_called_once_with(
                 search_results, search_criteria=criteria, **{cruncher: {}}
@@ -328,6 +329,41 @@ class TestEodagCli(unittest.TestCase):
                 **{cruncher: {"minimum_overlap": 10}}
             )
 
+    @mock.patch("eodag.cli.EODataAccessGateway", autospec=True)
+    def test_eodag_search_all(self, dag):
+        """Calling eodag search with --bbox argument valid"""
+        with self.user_conf() as conf_file:
+            product_type = "whatever"
+            self.runner.invoke(
+                eodag,
+                [
+                    "search",
+                    "--conf",
+                    conf_file,
+                    "-p",
+                    product_type,
+                    "-g",
+                    "POLYGON ((1 43, 1 44, 2 44, 2 43, 1 43))",
+                    "--all",
+                ],
+            )
+            api_obj = dag.return_value
+            api_obj.search_all.assert_called_once_with(
+                items_per_page=None,
+                startTimeFromAscendingNode=None,
+                completionTimeFromAscendingNode=None,
+                cloudCover=None,
+                geometry="POLYGON ((1 43, 1 44, 2 44, 2 43, 1 43))",
+                instrument=None,
+                platform=None,
+                platformSerialIdentifier=None,
+                processingLevel=None,
+                sensorType=None,
+                productType=product_type,
+                id=None,
+                locations=None,
+            )
+
     def test_eodag_list_product_type_ok(self):
         """Calling eodag list without provider return all supported product types"""
         all_supported_product_types = [
diff --git a/tests/units/test_core.py b/tests/units/test_core.py
index 221f70bc..4bde0171 100644
--- a/tests/units/test_core.py
+++ b/tests/units/test_core.py
@@ -520,6 +520,19 @@ class TestCoreSearch(unittest.TestCase):
         with self.assertRaises(NoMatchingProductType):
             self.dag.guess_product_type()
 
+    def test_guess_product_type_has_no_limit(self):
+        """guess_product_type must run a whoosh search without any limit"""
+        # Filter that should give more than 10 products referenced in the catalog.
+        opt_prods = [
+            p for p in self.dag.list_product_types() if p["sensorType"] == "OPTICAL"
+        ]
+        if len(opt_prods) <= 10:
+            self.skipTest("This test requires that more than 10 products are 'OPTICAL'")
+        guesses = self.dag.guess_product_type(
+            sensorType="OPTICAL",
+        )
+        self.assertGreater(len(guesses), 10)
+
     def test__prepare_search_no_parameters(self):
         """_prepare_search must create some kwargs even when no parameter has been provided"""  # noqa
         prepared_search = self.dag._prepare_search()