Repository: eWaterCycle/era5cli

Issue:
Add support for regional subselection
From the [CDS documentation](https://confluence.ecmwf.int/display/CKB/How+to+download+ERA5):

> Currently the interactive CDS forms don't allow users to perform a regional sub-selection or an interpolation of the selected data. However, both actions can be performed using the following CDS API keywords:
For a geographical area subset, use key area.
For a different grid resolution, use key grid.
Please note that the ERA5 native grid in CDS is 0.25°x0.25° (atmosphere), 0.5°x0.5° (ocean waves), Mean, spread and members: 0.5°x0.5° (atmosphere), 1°x1° (ocean waves).

We want to integrate this option in era5cli.

Discussion:
bvreede: Things to fix and add:
- [ ] Documentation for area options in info.py
- [ ] Include coordinates in output file name
- [ ] Fix cli tests currently set to xfail
- [ ] Fix Travis build
codecov[bot]: # [Codecov](https://codecov.io/gh/eWaterCycle/era5cli/pull/70?src=pr&el=h1) Report
> Merging [#70](https://codecov.io/gh/eWaterCycle/era5cli/pull/70?src=pr&el=desc) (37018e2) into [master](https://codecov.io/gh/eWaterCycle/era5cli/commit/39d16397e219e938d2332819f197efd775325e64?el=desc) (39d1639) will **decrease** coverage by `0.59%`.
> The diff coverage is `96.80%`.

[![Impacted file tree graph](https://codecov.io/gh/eWaterCycle/era5cli/pull/70/graphs/tree.svg?width=650&height=150&src=pr&token=qeZXgGASBK)](https://codecov.io/gh/eWaterCycle/era5cli/pull/70?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master      #70      +/-   ##
==========================================
- Coverage   98.06%   97.47%   -0.60%     
==========================================
  Files           7        7              
  Lines         310      356      +46     
==========================================
+ Hits          304      347      +43     
- Misses          6        9       +3     
```


| [Impacted Files](https://codecov.io/gh/eWaterCycle/era5cli/pull/70?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [era5cli/fetch.py](https://codecov.io/gh/eWaterCycle/era5cli/pull/70/diff?src=pr&el=tree#diff-ZXJhNWNsaS9mZXRjaC5weQ==) | `96.98% <96.42%> (-1.44%)` | :arrow_down: |
| [era5cli/cli.py](https://codecov.io/gh/eWaterCycle/era5cli/pull/70/diff?src=pr&el=tree#diff-ZXJhNWNsaS9jbGkucHk=) | `96.87% <100.00%> (+0.13%)` | :arrow_up: |
| [era5cli/info.py](https://codecov.io/gh/eWaterCycle/era5cli/pull/70/diff?src=pr&el=tree#diff-ZXJhNWNsaS9pbmZvLnB5) | `96.15% <100.00%> (+0.32%)` | :arrow_up: |
| [era5cli/inputref.py](https://codecov.io/gh/eWaterCycle/era5cli/pull/70/diff?src=pr&el=tree#diff-ZXJhNWNsaS9pbnB1dHJlZi5weQ==) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/eWaterCycle/era5cli/pull/70?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/eWaterCycle/era5cli/pull/70?src=pr&el=footer). Last update [39d1639...37018e2](https://codecov.io/gh/eWaterCycle/era5cli/pull/70?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

bvreede: Some things to pay attention to when reviewing:
- Is using SystemExit as an error type appropriate in testing (see test_cli.py)
- Because the area coordinates are floats, they always have decimal points. This looks ugly in particular in file names, especially when the original input was integers. Should this be addressed?

Patch:
diff --git a/README.rst b/README.rst
index 3078fdd..e0a0c30 100644
--- a/README.rst
+++ b/README.rst
@@ -33,9 +33,10 @@ With era5cli you can: ​
 - select multiple variables for several months and years
 - split outputs by years, producing a separate file for every year instead of merging them in one file
 - download multiple files at once
+- extract data for a sub-region of the globe
 - download ERA5 back extension (preliminary version)
 
 .. inclusion-marker-end-do-not-remove
 
-| Free software: Apache Software License 2.0 
+| Free software: Apache Software License 2.0
 | Documentation: https://era5cli.readthedocs.io
diff --git a/era5cli/cli.py b/era5cli/cli.py
index 4b3d65f..6991a97 100644
--- a/era5cli/cli.py
+++ b/era5cli/cli.py
@@ -159,6 +159,22 @@ def _build_parser():
                              ''')
     )
 
+    common.add_argument(
+        "--area", nargs=4, type=float,
+        required=False,
+        help=textwrap.dedent('''\
+                            Coordinates in case extraction of a subregion is
+                            requested. Specified as
+                            ymax xmin ymin xmax with x and y in the
+                            range -180, +180 and -90, +90, respectively
+                            e.g. --area 90 -180 -90 180.
+                            Requests are rounded down to two decimals.
+                            Without specification, the whole available area
+                            will be returned.
+
+                            ''')
+    )
+
     mnth = argparse.ArgumentParser(add_help=False)
 
     mnth.add_argument(
@@ -383,6 +399,7 @@ def _execute(args):
             days=days,
             hours=hours,
             variables=args.variables,
+            area=args.area,
             outputformat=args.format,
             outputprefix=args.outputprefix,
             period=args.command,
diff --git a/era5cli/fetch.py b/era5cli/fetch.py
index 4cd216e..2ff8fc8 100644
--- a/era5cli/fetch.py
+++ b/era5cli/fetch.py
@@ -26,6 +26,12 @@ class Fetch:
             hours to download data for.
         variables: list(str)
             List of variable names to download data for.
+        area: None, list(float)
+            Coordinates in case extraction of a subregion is requested.
+            Specified as [ymax, xmin, ymin, xmax], with x and y
+            in the range -180, +180 and -90, +90, respectively. Requests are
+            rounded down to two decimals. Without specification, the whole
+            available area will be returned.
         outputformat: str
             Type of file to download: 'netcdf' or 'grib'.
         outputprefix: str
@@ -65,7 +71,7 @@ class Fetch:
 
     def __init__(self, years: list, months: list, days: list,
                  hours: list, variables: list, outputformat: str,
-                 outputprefix: str, period: str, ensemble: bool,
+                 outputprefix: str, period: str, ensemble: bool, area=None,
                  statistics=None, synoptic=None, pressurelevels=None,
                  merge=False, threads=None, prelimbe=False, land=False):
         """Initialization of Fetch class."""
@@ -86,6 +92,9 @@ class Fetch:
         """list(int): List of pressure levels."""
         self.variables = variables
         """list(str): List of variables."""
+        self.area = area
+        """list(float): Coordinates specifying the subregion that will be
+        extracted. Default is None for whole available area."""
         self.outputformat = outputformat
         """str: File format of output file."""
         self.years = years
@@ -149,6 +158,13 @@ class Fetch:
             raise ValueError('Unknown outputformat: {}'.format(
                 self.outputformat))
 
+    def _process_areaname(self):
+        (ymax, xmin, ymin, xmax) = [round(c) for c in self.area]
+        def lon(x): return f"{x}E" if x >= 0 else f"{abs(x)}W"
+        def lat(y): return f"{y}N" if y >= 0 else f"{abs(y)}S"
+        name = f"_{lon(xmin)}-{lon(xmax)}_{lat(ymin)}-{lat(ymax)}"
+        return name
+
     def _define_outputfilename(self, var, years):
         """Define output filename."""
         start, end = years[0], years[-1]
@@ -157,6 +173,8 @@ class Fetch:
                   else self.outputprefix)
         yearblock = f"{start}-{end}" if not start == end else f"{start}"
         fname = f"{prefix}_{var}_{yearblock}_{self.period}"
+        if self.area:
+            fname += self._process_areaname()
         if self.ensemble:
             fname += "_ensemble"
         if self.statistics:
@@ -267,6 +285,30 @@ class Fetch:
                 "Invalid variable name: {}".format(variable)
             )
 
+    def _check_area(self):
+        """Confirm that area parameters are correct."""
+        (ymax, xmin, ymin, xmax) = self.area
+        if not (-90 <= ymax <= 90
+                and -90 <= ymin <= 90
+                and -180 <= xmin <= 180
+                and -180 <= xmax <= 180
+                and ymax > ymin
+                and xmax != xmin
+                ):
+            raise ValueError(
+                "Provide coordinates as ymax xmin ymin xmax. "
+                "x must be in range -180,+180 and y must be in range -90,+90."
+            )
+
+    def _parse_area(self):
+        """Parse area parameters to accepted coordinates."""
+        self._check_area()
+        area = [round(coord, ndigits=2) for coord in self.area]
+        if self.area != area:
+            print(
+                f"NB: coordinates {self.area} rounded down to two decimals.\n")
+        return area
+
     def _build_name(self, variable):
         """Build up name of dataset to use"""
 
@@ -310,6 +352,9 @@ class Fetch:
             self._check_levels()
             request["pressure_level"] = self.pressure_levels
 
+        if self.area:
+            request["area"] = self._parse_area()
+
         product_type = self._product_type()
         if product_type is not None:
             request["product_type"] = product_type


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 7ec3242..5b0fc84 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -28,6 +28,49 @@ def test_parse_args():
     assert not args.threads
     assert args.variables == ['total_precipitation']
     assert args.land
+    assert not args.area
+
+
+def test_area_argument():
+    """Test if area argument is parsed correctly."""
+    # Test if area arguments are parsed correctly
+    argv = ['hourly', '--startyear', '2008',
+            '--variables', 'total_precipitation', '--statistics',
+            '--endyear', '2008', '--ensemble',
+            '--area', '90', '-180', '-90', '180']
+    args = cli._parse_args(argv)
+    assert args.area == [90, -180, -90, 180]
+
+    # Check that area defaults to None
+    argv = ['hourly', '--startyear', '2008',
+            '--variables', 'total_precipitation', '--statistics',
+            '--endyear', '2008', '--ensemble']
+    args = cli._parse_args(argv)
+    assert not args.area
+
+    # Requires four values
+    with pytest.raises(SystemExit):
+        argv = ['hourly', '--startyear', '2008',
+                '--variables', 'total_precipitation', '--statistics',
+                '--endyear', '2008', '--ensemble',
+                '--area', '90', '-180', '-90']
+        cli._parse_args(argv)
+
+    # A value cannot be missing
+    with pytest.raises(SystemExit):
+        argv = ['hourly', '--startyear', '2008',
+                '--variables', 'total_precipitation', '--statistics',
+                '--endyear', '2008', '--ensemble',
+                '--area', '90', '-180', '-90', '']
+        cli._parse_args(argv)
+
+    # Values must be numeric
+    with pytest.raises(SystemExit):
+        argv = ['hourly', '--startyear', '2008',
+                '--variables', 'total_precipitation', '--statistics',
+                '--endyear', '2008', '--ensemble',
+                '--area', '90', '-180', '-90', 'E']
+        cli._parse_args(argv)
 
 
 def test_period_args():
diff --git a/tests/test_fetch.py b/tests/test_fetch.py
index ef18f26..701801e 100644
--- a/tests/test_fetch.py
+++ b/tests/test_fetch.py
@@ -7,7 +7,8 @@ import unittest.mock as mock
 
 def initialize(outputformat='netcdf', merge=False, statistics=None,
                synoptic=None, ensemble=True, pressurelevels=None,
-               threads=2, period='hourly', variables=['total_precipitation'],
+               threads=2, period='hourly', area=None,
+               variables=['total_precipitation'],
                years=[2008, 2009], months=list(range(1, 13)),
                days=list(range(1, 32)), hours=list(range(0, 24)),
                prelimbe=False, land=False):
@@ -16,6 +17,7 @@ def initialize(outputformat='netcdf', merge=False, statistics=None,
                        months=months,
                        days=days,
                        hours=hours,
+                       area=area,
                        variables=variables,
                        outputformat=outputformat,
                        outputprefix='era5',
@@ -201,6 +203,27 @@ def test_define_outputfilename():
     fn = 'era5-land_total_precipitation_2008_hourly.nc'
     assert fname == fn
 
+    era5.area = [90.0, -180.0, -90.0, 180.0]
+    fname = era5._define_outputfilename('total_precipitation', [2008])
+    fn = (
+        'era5-land_total_precipitation_2008_hourly_180W-180E_90S-90N.nc'
+    )
+    assert fname == fn
+
+    era5.area = [90.0, -180.0, -80.999, 170.001]
+    fname = era5._define_outputfilename('total_precipitation', [2008])
+    fn = (
+        'era5-land_total_precipitation_2008_hourly_180W-170E_81S-90N.nc'
+    )
+    assert fname == fn
+
+    era5.area = [0, 120, -90, 180]
+    fname = era5._define_outputfilename('total_precipitation', [2008])
+    fn = (
+        'era5-land_total_precipitation_2008_hourly_120E-180E_90S-0N.nc'
+    )
+    assert fname == fn
+
 
 def test_number_outputfiles(capsys):
     """Test function for the number of outputs."""
@@ -520,3 +543,58 @@ def test_more_incompatible_options():
     era5 = initialize(statistics=True, ensemble=False)
     with pytest.raises(ValueError):
         era5._build_request('total_precipitation', [2008])
+
+
+def test_area():
+    """Test that area is parsed properly."""
+    era5 = initialize()
+    assert era5.area is None
+
+    era5 = initialize(area=[90, -180, -90, 180])
+    (name, request) = era5._build_request('total_precipitation', [2008])
+    assert era5.area == [90, -180, -90, 180]
+    assert request["area"] == [90, -180, -90, 180]
+
+    # Decimals are rounded down
+    era5 = initialize(area=[89.9999, -179.90, -90.0000, 179.012])
+    (name, request) = era5._build_request('total_precipitation', [2008])
+    assert request["area"] == [90.0, -179.90, -90.0, 179.01]
+
+    # ymax may not be lower than ymin
+    with pytest.raises(ValueError):
+        era5 = initialize(area=[-10, -180, 10, 180])
+        era5._build_request('total_precipitation', [2008])
+
+    # xmin higher than xmax should be ok
+    era5 = initialize(area=[90, 120, -90, -120])
+    (name, request) = era5._build_request('total_precipitation', [2008])
+    assert request["area"] == [90.0, 120.0, -90.0, -120.0]
+
+    # ymax may not equal ymin
+    with pytest.raises(ValueError):
+        era5 = initialize(area=[0, -180, 0, 180])
+        era5._build_request('total_precipitation', [2008])
+
+    # xmin may not equal xmax
+    with pytest.raises(ValueError):
+        era5 = initialize(area=[90, 0, -90, 0])
+        era5._build_request('total_precipitation', [2008])
+
+    # ymax, xmin, ymin, xmax may not be out of bounds
+    with pytest.raises(ValueError):
+        era5 = initialize(area=[1000, -180, -90, 180])
+        era5._build_request('total_precipitation', [2008])
+    with pytest.raises(ValueError):
+        era5 = initialize(area=[90, 1000, -90, 180])
+        era5._build_request('total_precipitation', [2008])
+    with pytest.raises(ValueError):
+        era5 = initialize(area=[90, -180, 1000, 180])
+        era5._build_request('total_precipitation', [2008])
+    with pytest.raises(ValueError):
+        era5 = initialize(area=[90, -180, -90, 1000])
+        era5._build_request('total_precipitation', [2008])
+
+    # Coordinate missing
+    with pytest.raises(ValueError):
+        era5 = initialize(area=[-180, 180, -90])
+        era5._build_request('total_precipitation', [2008])