internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: opendatacube/datacube-core

Issue:
`compute_reproject_roi` incorrect behaviour for non-overlapping geoboxes
### Expected behaviour

When `src` and `dst` do no overlap I expect both `src_roi` and `dst_roi` to be empty on output. Seems like in some cases one can get empty `src_roi` while `dst_roi` isn't empty.

See user report here: https://github.com/opendatacube/datacube-stats/issues/173



Discussion:
Problem is in this function:

https://github.com/opendatacube/datacube-core/blob/ac2866b5b9cce7ee633799dc54ca532dd2a4a0cf/datacube/utils/geometry/tools.py#L600-L607

On line 602 `roi_src` can be empty, say it's a vertical line with zero width, but non-zero height. After projecting back to `dst` coordinate reference (line 605) vertical line becomes a slightly angled line, and so bounding box of that line is no longer zero width. So this explains how empty source roi can still result in non-empty dst roi.

The fix is to short circuit computation soon after line 602 and set `dst_roi` to some empty value like `r_[0:0,0:0]`.




Resolving patch:
diff --git a/datacube/utils/geometry/tools.py b/datacube/utils/geometry/tools.py
--- a/datacube/utils/geometry/tools.py
+++ b/datacube/utils/geometry/tools.py
@@ -601,6 +601,9 @@ def compute_roi(src, dst, tr, pts_per_side, padding, align):
         XY = np.vstack(tr.back(gbox_boundary(dst, pts_per_side)))
         roi_src = roi_from_points(XY, src.shape, padding, align=align)
 
+        if roi_is_empty(roi_src):
+            return (roi_src, np.s_[0:0, 0:0])
+
         # project src roi back into dst and compute roi from that
         xy = np.vstack(tr(roi_boundary(roi_src, pts_per_side)))
         roi_dst = roi_from_points(xy, dst.shape, padding=0)  # no need to add padding twice


Test patch:
diff --git a/tests/test_geometry.py b/tests/test_geometry.py
--- a/tests/test_geometry.py
+++ b/tests/test_geometry.py
@@ -20,6 +20,7 @@
     is_affine_st,
     apply_affine,
     compute_axis_overlap,
+    roi_is_empty,
     w_,
 )
 from datacube.testutils.geom import (
@@ -959,6 +960,31 @@ def test_compute_reproject_roi():
     assert roi_shape(rr.roi_dst) == src[roi_].shape
 
 
+def test_compute_reproject_roi_issue647():
+    """ In some scenarios non-overlapping geoboxes will result in non-empty
+    `roi_dst` even though `roi_src` is empty.
+
+    Test this case separately.
+    """
+    from datacube.utils.geometry import CRS
+
+    src = GeoBox(10980, 10980,
+                 Affine(10, 0, 300000,
+                        0, -10, 5900020),
+                 CRS('epsg:32756'))
+
+    dst = GeoBox(976, 976, Affine(10, 0, 1730240,
+                                  0, -10, -4170240),
+                 CRS('EPSG:3577'))
+
+    assert src.extent.overlaps(dst.extent.to_crs(src.crs)) is False
+
+    rr = compute_reproject_roi(src, dst)
+
+    assert roi_is_empty(rr.roi_src)
+    assert roi_is_empty(rr.roi_dst)
+
+
 def test_window_from_slice():
     from numpy import s_
Repository: opendatacube/datacube-core

Issue:
`unary_union` does not detect failures
In this function:

https://github.com/opendatacube/datacube-core/blob/0ab135ef9986bee0ea476ced76fad7b9c2f47328/datacube/utils/geometry/_base.py#L692-L695

https://github.com/opendatacube/datacube-core/blob/0ab135ef9986bee0ea476ced76fad7b9c2f47328/datacube/utils/geometry/_base.py#L711-L712

`ogr.UnionCascaded()` can return `None` on failure, this `None` is then passed on to `Geometry` constructor (`._geom` property), but `Geometry` class assumes that `_geom` is not `None`. So there is no clean way to check if union succeeded in the first place(without accessing private members), probably `unary_union` should just return `None` on failure to mimic behaviour of the underlying function.

 


Discussion:

Resolving patch:
diff --git a/datacube/utils/geometry/_base.py b/datacube/utils/geometry/_base.py
--- a/datacube/utils/geometry/_base.py
+++ b/datacube/utils/geometry/_base.py
@@ -303,6 +303,8 @@ def _get_coordinates(geom):
 
 
 def _make_geom_from_ogr(geom, crs):
+    if geom is None:
+        return None
     result = Geometry.__new__(Geometry)
     result._geom = geom  # pylint: disable=protected-access
     result.crs = crs


Test patch:
diff --git a/tests/test_geometry.py b/tests/test_geometry.py
--- a/tests/test_geometry.py
+++ b/tests/test_geometry.py
@@ -199,6 +199,8 @@ def test_unary_union():
     assert union4.type == 'Polygon'
     assert union4.area == 2.5 * box1.area
 
+    assert geometry.unary_union([]) is None
+
     with pytest.raises(ValueError):
         pt = geometry.point(6, 7, epsg4326)
         geometry.unary_union([pt, pt])
Repository: opendatacube/datacube-core

Issue:
Errors when running against the impending sqlalchemy 1.3 release (in beta)
Originally reported in #667

Datacube consistently fails when run against the current beta version of sqlalchemy. [According to](https://www.sqlalchemy.org/blog/2019/02/08/sqlalchemy-1.3.0b3-released/) the sqlalchemy devs this release "1.3b3 should hopefully be the last beta release for 1.3, as no additional major changes are planned."

This isn't currently a problem, but it will break all of our builds and guides if not resolved before 1.3 is declared stable.

Manually reproduce the error with:

```
    pip install sqlalchemy==1.3b3
    datacube system init
```

- Either the sqlalchemy 1.3 beta has a bug, which we should report to them.
- Or our own code is doing something incorrect and we should fix it before 1.3 is declared stable.

Errors when running against the impending sqlalchemy 1.3 release (in beta)
Originally reported in #667

Datacube consistently fails when run against the current beta version of sqlalchemy. [According to](https://www.sqlalchemy.org/blog/2019/02/08/sqlalchemy-1.3.0b3-released/) the sqlalchemy devs this release "1.3b3 should hopefully be the last beta release for 1.3, as no additional major changes are planned."

This isn't currently a problem, but it will break all of our builds and guides if not resolved before 1.3 is declared stable.

Manually reproduce the error with:

```
    pip install sqlalchemy==1.3b3
    datacube system init
```

- Either the sqlalchemy 1.3 beta has a bug, which we should report to them.
- Or our own code is doing something incorrect and we should fix it before 1.3 is declared stable.



Discussion:

Resolving patch:
diff --git a/datacube/drivers/postgres/sql.py b/datacube/drivers/postgres/sql.py
--- a/datacube/drivers/postgres/sql.py
+++ b/datacube/drivers/postgres/sql.py
@@ -57,7 +57,11 @@ class CommonTimestamp(GenericFunction):
     package = 'agdc'
     identifier = 'common_timestamp'
 
-    name = '%s.common_timestamp' % SCHEMA_NAME
+    name = 'common_timestamp'
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.packagenames = ['%s' % SCHEMA_NAME]
 
 
 # pylint: disable=too-many-ancestors
@@ -66,7 +70,11 @@ class Float8Range(GenericFunction):
     package = 'agdc'
     identifier = 'float8range'
 
-    name = '%s.float8range' % SCHEMA_NAME
+    name = 'float8range'
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.packagenames = ['%s' % SCHEMA_NAME]
 
 
 class PGNAME(sqltypes.Text):


Test patch:
Repository: opendatacube/datacube-core

Issue:
Address future warning from upcoming xarray datetime changes
Running `develop` branch against recent `xarray` generates following `FutureWarning`:
 
```
xarray/core/variable.py:134: FutureWarning: Converting timezone-aware DatetimeArray to ...
   ...timezone-naive ndarray with 'datetime64[ns]' dtype. ...
   ...In the future, this will return an ndarray with 'object' dtype where each element...
   ...is a 'pandas.Timestamp' with the correct 'tz'.
	To accept the future behavior, pass 'dtype=object'.
	To keep the old behavior, pass 'dtype="datetime64[ns]"'.
  return np.asarray(pd.Series(values.ravel())).reshape(values.shape)
```

I guess this is generated from `create_storage` or `group_datasets` whenever constructing time axis.



Discussion:
Hopefully this makes the _datetime conversion dance (tm)_ I've had to do with xarray less painful in the future.

Good catch on the warning.
Warning comes from converting `ds.center_time` to `numpy.datetime64`, `center_time` includes time zone info, I have only seen it being UTC, but I guess this depends on what is indexed in the database as there is nothing in the code that modifies timestamp stored in metadata that I can see. I guess the correct fix for this is to convert `center_time` to UTC, then strip tzinfo then convert that to `np.datetime64[ns]`, this would have to happen in here:

https://github.com/opendatacube/datacube-core/blob/c33012db799b5ae38277c53f4c2f934a03044e74/datacube/api/query.py#L164-L172





Resolving patch:
diff --git a/datacube/api/core.py b/datacube/api/core.py
--- a/datacube/api/core.py
+++ b/datacube/api/core.py
@@ -1,6 +1,7 @@
 import uuid
 from itertools import groupby
 from typing import Union, Optional, Dict, Tuple
+import datetime
 
 import numpy
 import xarray
@@ -365,11 +366,20 @@ def group_datasets(datasets, group_by):
         def ds_sorter(ds):
             return sort_key(ds), getattr(ds, 'id', 0)
 
+        def norm_axis_value(x):
+            if isinstance(x, datetime.datetime):
+                # For datetime we convert to UTC, then strip timezone info
+                # to avoid numpy/pandas warning about timezones
+                if x.tzinfo is not None:
+                    x = x.astimezone(datetime.timezone.utc).replace(tzinfo=None)
+                return numpy.datetime64(x, 'ns')
+            return x
+
         def mk_group(group):
             dss = tuple(sorted(group, key=ds_sorter))
             # TODO: decouple axis_value from group sorted order
             axis_value = sort_key(dss[0])
-            return (axis_value, dss)
+            return (norm_axis_value(axis_value), dss)
 
         datasets = sorted(datasets, key=group_func)
 


Test patch:
Repository: opendatacube/datacube-core

Issue:
Vitrual product aggregation needs to carry the temporal coordinates attributes
### Expected behaviour
The output time coordinates from `aggregation` should have the exact same attributes as the input product.

### Actual behaviour
Only the values of the time coordinates present in the output product.


### Steps to reproduce the behaviour
Any time reduction aggregation.

### Environment information
N/A



Discussion:

Resolving patch:
diff --git a/datacube/virtual/impl.py b/datacube/virtual/impl.py
--- a/datacube/virtual/impl.py
+++ b/datacube/virtual/impl.py
@@ -431,7 +431,9 @@ def statistic(coords, value):
             return result
 
         groups = list(xr_map(grouped.pile, statistic))
-        return xarray.concat(groups, dim=dim).assign_attrs(**select_unique([g.attrs for g in groups]))
+        result = xarray.concat(groups, dim=dim).assign_attrs(**select_unique([g.attrs for g in groups]))
+        result.coords[dim].attrs.update(grouped.pile[dim].attrs)
+        return result
 
 
 class Collate(VirtualProduct):


Test patch:
Repository: opendatacube/datacube-core

Issue:
`find_datasets` returns empty result set when `lon=(-180,180), lat=(-90,90)`
### Expected behaviour

```python
len(dc.find_datasets(product=p, time=t)) == len(
    dc.find_datasets(product=p, time=t, 
    lon=(-180,180), lat=(-90, 90))
```


### Actual behaviour

```python
len(
    dc.find_datasets(product=p, time=t, 
    lon=(-180,180), lat=(-90, 90)) == 0
```

Problem is likely in post query filtering of the result set:

https://github.com/opendatacube/datacube-core/blob/69e1a3b486f5f34c3d863a611efa65979b92f7eb/datacube/api/core.py#L706-L711

This function assumes that query polygon can be safely transformed into CRS of the dataset, this breaks down for "whole earth in EPSG4326" polygon.






Discussion:

Resolving patch:
diff --git a/datacube/api/core.py b/datacube/api/core.py
--- a/datacube/api/core.py
+++ b/datacube/api/core.py
@@ -706,8 +706,9 @@ def output_geobox(like=None, output_crs=None, resolution=None, align=None,
 def select_datasets_inside_polygon(datasets, polygon):
     # Check against the bounding box of the original scene, can throw away some portions
     assert polygon is not None
+    query_crs = polygon.crs
     for dataset in datasets:
-        if intersects(polygon.to_crs(dataset.crs), dataset.extent):
+        if intersects(polygon, dataset.extent.to_crs(query_crs)):
             yield dataset
 
 


Test patch:
Repository: opendatacube/datacube-core

Issue:
Dataset `__eq__` fails when other object is not a Dataset
### Expected behaviour

```python
ds, *_ = dc.find_datasets(..)
assert (ds == "33") is False
```


### Actual behaviour

Error is raised inside `__eq__` operator.

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-a8ddf445deb8> in <module>
      1 ds, *_ = dc.find_datasets(product=product, limit=10)
      2 
----> 3 assert (ds == "33") is False

~/wk/datacube-core/datacube/model/__init__.py in __eq__(self, other)
    286 
    287     def __eq__(self, other) -> bool:
--> 288         return self.id == other.id
    289 
    290     def __hash__(self):

AttributeError: 'str' object has no attribute 'id'
```







Discussion:

Resolving patch:
diff --git a/datacube/model/__init__.py b/datacube/model/__init__.py
--- a/datacube/model/__init__.py
+++ b/datacube/model/__init__.py
@@ -285,7 +285,9 @@ def xytuple(obj):
         return None
 
     def __eq__(self, other) -> bool:
-        return self.id == other.id
+        if isinstance(other, Dataset):
+            return self.id == other.id
+        return False
 
     def __hash__(self):
         return hash(self.id)


Test patch:
diff --git a/tests/test_model.py b/tests/test_model.py
--- a/tests/test_model.py
+++ b/tests/test_model.py
@@ -45,6 +45,14 @@ def test_gridspec_upperleft():
     assert cells[(30, 15)].extent.boundingbox == tile_bbox
 
 
+def test_dataset_basics():
+    ds = mk_sample_dataset([dict(name='a')])
+    assert ds == ds
+    assert ds != "33"
+    assert (ds == "33") is False
+    assert str(ds) == repr(ds)
+
+
 def test_dataset_measurement_paths():
     format = 'GeoTiff'
Repository: opendatacube/datacube-core

Issue:
Use of unescaped strings for generating DB queries
User management functions in `/drivers/postgres/_api.py` use `String.format` instead of relying on safer query building formatting provided by `connection.execute`

https://github.com/opendatacube/datacube-core/blob/5f7211d2eb26ac10d6dfb754418d4bd3079cdf7a/datacube/drivers/postgres/_api.py#L878-L881


Discussion:
I can take this one for the Bug Squash.

Resolving patch:
diff --git a/datacube/drivers/postgres/_api.py b/datacube/drivers/postgres/_api.py
--- a/datacube/drivers/postgres/_api.py
+++ b/datacube/drivers/postgres/_api.py
@@ -1016,20 +1016,19 @@ def list_users(self):
     def create_user(self, username, password, role, description=None):
         pg_role = _core.to_pg_role(role)
         username = escape_pg_identifier(self._connection, username)
-        self._connection.execute(
-            'create user {username} password %s in role {role}'.format(username=username, role=pg_role),
-            password
-        )
+        sql = text('create user {username} password :password in role {role}'.format(username=username, role=pg_role))
+        self._connection.execute(sql,
+                                 password=password)
         if description:
-            self._connection.execute(
-                'comment on role {username} is %s'.format(username=username), description
-            )
+            sql = text('comment on role {username} is :description'.format(username=username))
+            self._connection.execute(sql,
+                                     description=description)
 
     def drop_users(self, users):
         # type: (Iterable[str]) -> None
         for username in users:
-            self._connection.execute('drop role {username}'.format(
-                username=escape_pg_identifier(self._connection, username)))
+            sql = text('drop role {username}'.format(username=escape_pg_identifier(self._connection, username)))
+            self._connection.execute(sql)
 
     def grant_role(self, role, users):
         # type: (str, Iterable[str]) -> None


Test patch:
Repository: opendatacube/datacube-core

Issue:
If DB_PORT is not set, config process sets port to an empty string
I have an existing environment that sets up the datacube connection using this:
```
      - DB_HOSTNAME=host.docker.internal
      - DB_USERNAME=opendatacube
      - DB_PASSWORD=opendatacubepassword
      - DB_DATABASE=opendatacube
```

and with the new changes to read config from environment variables over the config file, the port is required to be set with `DB_PORT=5432`.

Expected behaviour was that if the port is blank it is set to the default for Postgres.

https://github.com/opendatacube/datacube-core/blob/8481d907b198a1c8946326b8b70625a9a8523a12/datacube/config.py#L265


Discussion:
@alexgleith is it not set or set to an empty string?


Resolving patch:
diff --git a/datacube/config.py b/datacube/config.py
--- a/datacube/config.py
+++ b/datacube/config.py
@@ -233,7 +233,7 @@ def parse_env_params() -> Dict[str, str]:
               for k in DB_KEYS}
     return {k: v
             for k, v in params.items()
-            if v is not None}
+            if v is not None and v != ""}
 
 
 def _cfg_from_env_opts(opts: Dict[str, str],


Test patch:
diff --git a/tests/test_config.py b/tests/test_config.py
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -164,6 +164,16 @@ def check_env(**kw):
                          username='user',
                          password='pass@')
 
+    assert check_env(DB_DATABASE='db',
+                     DB_HOSTNAME='host.tld',
+                     DB_USERNAME='user',
+                     DB_PORT='',
+                     DB_PASSWORD='pass@') == dict(
+                         database='db',
+                         hostname='host.tld',
+                         username='user',
+                         password='pass@')
+
 
 def test_cfg_from_env(monkeypatch):
     def set_env(**kw):
Repository: opendatacube/datacube-core

Issue:
Documentation for indexing from s3 contains mistakes

resolution for EPSG:4326 should be in degrees not in meters:

https://github.com/opendatacube/datacube-core/commit/363a11c9f39a40c8fba958cb265ace193d7849b6#diff-95fd54d5e1fd0aea8de7aacba3ad495cR323




Discussion:

Resolving patch:
diff --git a/datacube/config.py b/datacube/config.py
--- a/datacube/config.py
+++ b/datacube/config.py
@@ -227,7 +227,7 @@ def auto_config() -> str:
     Render config to $DATACUBE_CONFIG_PATH or ~/.datacube.conf, but only if doesn't exist.
 
     option1:
-      DATACUBE_DB_URL  postgresql://user:password@host/database
+      DATACUBE_DB_URL  postgresql://user:password@host:port/database
 
     option2:
       DB_{HOSTNAME|PORT|USERNAME|PASSWORD|DATABASE}


Test patch:
Repository: opendatacube/datacube-core

Issue:
Update release process documentation
Many steps described in the document have since been automated, documentation should reflect that:

- Upload to pypi is done by Travis
- Updates for conda-forge are done by some bot that creates PR


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -32,7 +32,7 @@
 
 setup(
     name='datacube',
-    python_requires='>=3.5.2',
+    python_requires='>=3.6.0',
 
     url='https://github.com/opendatacube/datacube-core',
     author='Open Data Cube',
@@ -55,8 +55,8 @@
         "Operating System :: Microsoft :: Windows",
         "Programming Language :: Python",
         "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.5",
         "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
         "Topic :: Scientific/Engineering :: GIS",
         "Topic :: Scientific/Engineering :: Information Analysis",
     ],


Test patch:
Repository: opendatacube/datacube-core

Issue:
Virtual product aggregate dimension bug
### Expected behaviour
This recipe:

```yaml
            aggregate: xarray_reduction
            method: sum
            group_by: year
            input:
                product: ls5_nbart_geomedian_annual
                measurements: [red]
```
should be a no-op.

### Actual behaviour
```
ValueError: coordinate 'time' is a DataArray dimension, but it has shape () rather than expected shape (1,) matching the dimension size
````

This is caused by inconsistent behaviour of `xarray.DataArray.groupby` which drops the time dimension if there is only one dataset in the group.

### Environment information
* Which ``datacube --version`` are you using?
Commit aad9f10c99aa74178d21526e9e7ec051036e3533.



Discussion:

Resolving patch:
diff --git a/datacube/virtual/impl.py b/datacube/virtual/impl.py
--- a/datacube/virtual/impl.py
+++ b/datacube/virtual/impl.py
@@ -530,7 +530,7 @@ def to_box(value):
                                                        geopolygon=grouped.geopolygon)],
                                     dims=['_fake_'])
 
-        result = grouped.box.groupby(self['group_by'](grouped.box[dim])).apply(to_box).squeeze('_fake_')
+        result = grouped.box.groupby(self['group_by'](grouped.box[dim]), squeeze=False).apply(to_box).squeeze('_fake_')
         result[dim].attrs.update(grouped.box[dim].attrs)
 
         return VirtualDatasetBox(result, grouped.geobox, grouped.load_natively, grouped.product_definitions,


Test patch:
Repository: opendatacube/datacube-core

Issue:
Leaking DB connections in intergration tests
Integration tests started occasionally failing with the following error:

```
psycopg2.OperationalError: FATAL:  sorry, too many clients already
```

I have not been able to reproduce this in the local dev environment, but it happens on Github Actions from time to time, typically when running ` integration_tests/test_dataset_add.py` suite.

Integration tests run  `datacube` CLI app, but in a fancy way, avoiding launching a separate process. This is needed for accurate coverage tracking, but I feel like this leads to resource leakage. Apps are designed for once off running, so they might not care about closing db connections in a "proper way".

Acceptable solutions:

- Bump connection pool to avoid problems
- Find a way for test fixtures to cleanup properly




Discussion:

Resolving patch:
diff --git a/datacube/ui/click.py b/datacube/ui/click.py
--- a/datacube/ui/click.py
+++ b/datacube/ui/click.py
@@ -215,16 +215,21 @@ def decorate(f):
         def with_index(local_config: config.LocalConfig,
                        *args,
                        **kwargs):
-            ctx = click.get_current_context()
+            command_path = click.get_current_context().command_path
             try:
                 index = index_connect(local_config,
-                                      application_name=app_name or ctx.command_path,
+                                      application_name=app_name or command_path,
                                       validate_connection=expect_initialised)
-                ctx.obj['index'] = index
                 _LOG.debug("Connected to datacube index: %s", index)
-                return f(index, *args, **kwargs)
             except (OperationalError, ProgrammingError) as e:
                 handle_exception('Error Connecting to database: %s', e)
+                return
+
+            try:
+                return f(index, *args, **kwargs)
+            finally:
+                index.close()
+                del index
 
         return functools.update_wrapper(with_index, f)
 


Test patch: