Repository: eWaterCycle/era5cli

Issue:
Add integration tests
At the moment, we test era5cli in two ways:
- Test if arguments given on the command line are parsed correctly and entered correctly to the Fetch class (`test_cli.py`)
- Test if information given to the Fetch class result in a correct CDS query (`test_fetch.py`).

It would be good to also add integration tests that combine these two, where arguments are given and the final query is asserted. This would make it easier to write tests for the tool to assert correct behavior of the tool as a whole.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/eWaterCycle/era5cli/pull/102?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle) Report
> Merging [#102](https://codecov.io/gh/eWaterCycle/era5cli/pull/102?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle) (088348f) into [master](https://codecov.io/gh/eWaterCycle/era5cli/commit/2240bc1e2d78a8a4e6a375daa89d8acc6609348e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle) (2240bc1) will **increase** coverage by `0.62%`.
> The diff coverage is `97.72%`.

[![Impacted file tree graph](https://codecov.io/gh/eWaterCycle/era5cli/pull/102/graphs/tree.svg?width=650&height=150&src=pr&token=qeZXgGASBK&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle)](https://codecov.io/gh/eWaterCycle/era5cli/pull/102?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle)

```diff
@@            Coverage Diff             @@
##           master     #102      +/-   ##
==========================================
+ Coverage   98.32%   98.94%   +0.62%     
==========================================
  Files           7        7              
  Lines         358      380      +22     
==========================================
+ Hits          352      376      +24     
+ Misses          6        4       -2     
```


| [Impacted Files](https://codecov.io/gh/eWaterCycle/era5cli/pull/102?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle) | Coverage Δ | |
|---|---|---|
| [era5cli/cli.py](https://codecov.io/gh/eWaterCycle/era5cli/pull/102/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle#diff-ZXJhNWNsaS9jbGkucHk=) | `98.03% <87.50%> (+1.16%)` | :arrow_up: |
| [era5cli/\_\_version\_\_.py](https://codecov.io/gh/eWaterCycle/era5cli/pull/102/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle#diff-ZXJhNWNsaS9fX3ZlcnNpb25fXy5weQ==) | `100.00% <100.00%> (ø)` | |
| [era5cli/fetch.py](https://codecov.io/gh/eWaterCycle/era5cli/pull/102/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle#diff-ZXJhNWNsaS9mZXRjaC5weQ==) | `99.45% <100.00%> (+0.65%)` | :arrow_up: |
| [era5cli/inputref.py](https://codecov.io/gh/eWaterCycle/era5cli/pull/102/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle#diff-ZXJhNWNsaS9pbnB1dHJlZi5weQ==) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/eWaterCycle/era5cli/pull/102?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/eWaterCycle/era5cli/pull/102?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle). Last update [ad5508a...088348f](https://codecov.io/gh/eWaterCycle/era5cli/pull/102?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=eWaterCycle).

bvreede: Some notes:
- I think the way calls and results are now mapped to the test can be improved, but I don't quite know how. One issue is that they need to be explicitly added to the `call_result` tuple, which may be forgotten and then not throw an error. A lesser point would be that moving around tests is not possible without renaming. (e.g. call1 and call3 should follow each other, but in order to do that, call2 and call3 need to be renamed as well as moved...)
- codecov fails because that one line extra in `main()` >.<
bvreede: Thanks to @sverhoeven and @SarahAlidoost!  :)
bvreede: Great suggestion, @sverhoeven, especially since it unveiled a missing warning message in the test!


Patch:
diff --git a/era5cli/cli.py b/era5cli/cli.py
index fadea62..f7cb0d7 100644
--- a/era5cli/cli.py
+++ b/era5cli/cli.py
@@ -425,10 +425,12 @@ def _execute(args):
         return True
 
 
-def main():
+def main(argv=None):
     """Main."""
     # get arguments
-    args = _parse_args(sys.argv[1:])
+    if argv is None:
+        argv = sys.argv
+    args = _parse_args(argv[1:])
     _execute(args)
 
 


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 6390f32..1184ed1 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -1,4 +1,4 @@
-"""Tests for era5cli utility functios."""
+"""Tests for era5cli utility functions."""
 
 import unittest.mock as mock
 import pytest
diff --git a/tests/test_integration.py b/tests/test_integration.py
new file mode 100644
index 0000000..1055dbc
--- /dev/null
+++ b/tests/test_integration.py
@@ -0,0 +1,128 @@
+"""Tests to check the full era5cli workflow."""
+
+import logging
+import pytest
+
+from textwrap import dedent
+
+from era5cli.cli import main
+
+
+# combine calls with result and possible warning message
+call_result = [
+    {
+        # orography is translated to geopotential in the query
+        "call": dedent("""\
+            era5cli hourly --variables orography --startyear 2008 --dryrun
+            """),
+        "result": dedent("""\
+            reanalysis-era5-single-levels {'variable': 'geopotential', 'year':
+            2008, 'month': ['01', '02', '03', '04', '05', '06', '07', '08',
+            '09', '10', '11', '12'], 'time': ['00:00', '01:00', '02:00',
+            '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00',
+            '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00',
+            '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00'],
+            'format': 'netcdf', 'product_type': 'reanalysis', 'day': ['01',
+            '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12',
+            '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23',
+            '24', '25', '26', '27', '28', '29', '30', '31']}
+            era5_orography_2008_hourly.nc"""),
+        "warn": "The variable 'orography' has been deprecated by CDS."
+    },
+    {
+        # geopotential needs '--levels surface' to be correctly interpreted
+        "call": dedent("""\
+            era5cli hourly --variables geopotential --startyear 2008 --dryrun
+            --levels surface"""),
+        "result": dedent("""\
+            reanalysis-era5-single-levels {'variable': 'geopotential', 'year':
+            2008, 'month': ['01', '02', '03', '04', '05', '06', '07', '08',
+            '09', '10', '11', '12'], 'time': ['00:00', '01:00', '02:00',
+            '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00',
+            '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00',
+            '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00'],
+            'format': 'netcdf', 'product_type': 'reanalysis', 'day': ['01',
+            '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12',
+            '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23',
+            '24', '25', '26', '27', '28', '29', '30', '31']}
+            era5_geopotential_2008_hourly.nc"""),
+        "warn": "Getting variable from surface level data."
+    },
+    {
+        # without --levels surface, geopotential calls pressure level data
+        "call": dedent("""\
+            era5cli hourly --variables geopotential --startyear 2008
+            --dryrun"""),
+        "result": dedent("""\
+            reanalysis-era5-pressure-levels {'variable': 'geopotential',
+            'year': 2008, 'month': ['01', '02', '03', '04', '05', '06', '07',
+            '08', '09', '10', '11', '12'], 'time': ['00:00', '01:00', '02:00',
+            '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00',
+            '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00',
+            '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00'],
+            'format': 'netcdf', 'pressure_level': [1, 2, 3, 5, 7, 10, 20, 30,
+            50, 70, 100, 125, 150, 175, 200, 225, 250, 300, 350, 400, 450, 500,
+            550, 600, 650, 700, 750, 775, 800, 825, 850, 875, 900, 925, 950,
+            975, 1000], 'product_type': 'reanalysis', 'day': ['01', '02', '03',
+            '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14',
+            '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25',
+            '26', '27', '28', '29', '30', '31']}
+            era5_geopotential_2008_hourly.nc"""),
+        "warn": "Getting variable from pressure level data."
+    },
+    {
+        # preliminary-back-extension is combined with monthly-means
+        "call": dedent("""\
+            era5cli monthly --variables temperature --startyear 1960 --prelimbe
+            --dryrun"""),
+        "result": dedent("""\
+            reanalysis-era5-pressure-levels-monthly-means-preliminary-back-extension
+            {'variable': 'temperature', 'year': 1960, 'month': ['01', '02',
+            '03', '04', '05', '06', '07', '08', '09', '10', '11', '12'],
+            'time': ['00:00'], 'format': 'netcdf', 'pressure_level': [1, 2, 3,
+            5, 7, 10, 20, 30, 50, 70, 100, 125, 150, 175, 200, 225, 250, 300,
+            350, 400, 450, 500, 550, 600, 650, 700, 750, 775, 800, 825, 850,
+            875, 900, 925, 950, 975, 1000], 'product_type':
+            'reanalysis-monthly-means-of-daily-means'}
+            era5_temperature_1960_monthly.nc""")
+    },
+    {
+        # era5-Land is combined with monthly means
+        "call": dedent("""\
+            era5cli monthly --variables snow_cover --startyear 2008 --land
+            --dryrun"""),
+        "result": dedent("""\
+            reanalysis-era5-land-monthly-means {'variable': 'snow_cover',
+            'year': 2008, 'month': ['01', '02', '03', '04', '05', '06', '07',
+            '08', '09', '10', '11', '12'], 'time': ['00:00'], 'format':
+            'netcdf', 'product_type': 'monthly_averaged_reanalysis'}
+            era5-land_snow_cover_2008_monthly.nc""")
+    }
+]
+
+
+def clean_ids(call):
+    call = call.replace('\n', ' ')
+    call = call.replace('--dryrun', '')
+    return(call)
+
+
+ids = [clean_ids(item["call"]) for item in call_result]
+
+
+@pytest.mark.parametrize("call_result", call_result, ids=ids)
+def test_main(call_result, capsys, caplog):
+    call = call_result["call"].split()
+    result = call_result["result"].replace('\n', ' ') + '\n'
+    # until the actual fetch is monkeypatched, make sure the tests are dryruns
+    if '--dryrun' not in call:
+        pytest.fail('call must be a dryrun')
+    with caplog.at_level(logging.INFO):
+        main(call)
+    captured = capsys.readouterr().out
+    assert result == captured
+    try:
+        warn = call_result["warn"]
+        assert warn in caplog.text
+    except KeyError:
+        assert caplog.text == ''