internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: opendatacube/datacube-core

Issue:
Skip include lineage verification for eo3 dataset documents
### Expected behaviour

eo3 lineages of the format 

```lineage: {"name": ["UUID"]}```

Should not be verified because they don't contain information about the linked dataset document. There is a work around by adding the `--no-verify-lineage` to `datacube dataset add  ...` but this should happen by default.

### Actual behaviour

The data provided in the lineage is compared against the linked dataset document which ends up comparing:

```python
{'id': UUID', 'lineage': {'source_datasets': {}}}
```

To the full dataset document and throws this error because the two aren't equal.

```ERROR Inconsistent lineage dataset```.

### Steps to reproduce the behaviour

Add a lineage to an eo3 dataset document and run:

```
datacube dataset add ...
```

### Environment information

* Which datacube --version are you using? Open Data Cube core, version 1.8.0
* What datacube deployment/enviornment are you running against? Local datacube deployment using the setup as in https://datacube-core.readthedocs.io/en/latest/ops/db_setup.html 


Discussion:
@Kirill888 I'm keen to have a go at this! can you assign me?
https://github.com/opendatacube/datacube-core/blob/0bf503fe5b92acbd8125b042de8b7123866a73a6/datacube/index/hl.py#L155-L163

I'd say in code above, even if `verify_lineage` is enabled, we should skip the check if `main_ds` is of EO3 kind.
Thanks for looking into it.

Resolving patch:
diff --git a/datacube/index/hl.py b/datacube/index/hl.py
--- a/datacube/index/hl.py
+++ b/datacube/index/hl.py
@@ -9,7 +9,7 @@
 from datacube.utils import changes, InvalidDocException, SimpleDocNav, jsonify_document
 from datacube.model.utils import dedup_lineage, remap_lineage_doc, flatten_datasets
 from datacube.utils.changes import get_doc_changes
-from .eo3 import prep_eo3
+from .eo3 import prep_eo3, is_doc_eo3
 
 
 class BadMatch(Exception):
@@ -152,7 +152,7 @@ def resolve(main_ds, uri):
         if missing_lineage and fail_on_missing_lineage:
             return None, "Following lineage datasets are missing from DB: %s" % (','.join(missing_lineage))
 
-        if verify_lineage:
+        if verify_lineage and not is_doc_eo3(main_ds.doc):
             bad_lineage = []
 
             for uuid in lineage_uuids:


Test patch:
diff --git a/integration_tests/test_dataset_add.py b/integration_tests/test_dataset_add.py
--- a/integration_tests/test_dataset_add.py
+++ b/integration_tests/test_dataset_add.py
@@ -255,7 +255,7 @@ def test_dataset_add(dataset_add_configs, index_empty, clirunner):
     assert 'WARNING --auto-match option is deprecated' in r.output
 
     # test dataset add eo3
-    r = clirunner(['dataset', 'add', '--no-verify-lineage', p.datasets_eo3])
+    r = clirunner(['dataset', 'add', p.datasets_eo3])
     assert r.exit_code == 0
 
     ds_eo3 = load_dataset_definition(p.datasets_eo3)
Repository: opendatacube/datacube-core

Issue:
Can't create credentialised s3_client if AWS_UNSIGNED is set
When running with `AWS_UNSIGNED=yes`, `.utils.aws.s3_client(creds=creds)` will ignore supplied credentials even though it should not.


Discussion:

Resolving patch:
diff --git a/datacube/utils/aws/__init__.py b/datacube/utils/aws/__init__.py
--- a/datacube/utils/aws/__init__.py
+++ b/datacube/utils/aws/__init__.py
@@ -268,7 +268,10 @@ def s3_client(profile: Optional[str] = None,
 
     """
     if aws_unsigned is None:
-        aws_unsigned = _aws_unsigned_check_env()
+        if creds is None:
+            aws_unsigned = _aws_unsigned_check_env()
+        else:
+            aws_unsigned = False
 
     if aws_unsigned:
         cfg.update(signature_version=botocore.UNSIGNED)
diff --git a/datacube/utils/dask.py b/datacube/utils/dask.py
--- a/datacube/utils/dask.py
+++ b/datacube/utils/dask.py
@@ -239,7 +239,8 @@ def pmap(func: Any,
 
 
 def _save_blob_to_file(data: Union[bytes, str],
-                       fname: str) -> Tuple[str, bool]:
+                       fname: str,
+                       with_deps=None) -> Tuple[str, bool]:
     if isinstance(data, str):
         data = data.encode('utf8')
 
@@ -257,8 +258,10 @@ def _save_blob_to_s3(data: Union[bytes, str],
                      profile: Optional[str] = None,
                      creds: Optional[ReadOnlyCredentials] = None,
                      region_name: Optional[str] = None,
+                     with_deps=None,
                      **kw) -> Tuple[str, bool]:
     from botocore.errorfactory import ClientError
+
     try:
         s3 = s3_client(profile=profile,
                        creds=creds,
@@ -302,7 +305,7 @@ def save_blob_to_file(data,
        the same path as calling code.
 
     """
-    return _save_blob_to_file_delayed(data, fname)
+    return _save_blob_to_file_delayed(data, fname, with_deps=with_deps)
 
 
 def save_blob_to_s3(data,
@@ -335,4 +338,5 @@ def save_blob_to_s3(data,
                                     profile=profile,
                                     creds=creds,
                                     region_name=region_name,
+                                    with_deps=with_deps,
                                     **kw)


Test patch:
Repository: opendatacube/datacube-core

Issue:
Feature request: write_cog to explicitly define no_data_values and dtypes
### Expected behaviour
`write_cog(ds, "output_file.tif",  dtype='float32', nodata=nan)`

### Actual behaviour
`write_cog(ds, "output_file.tif")`
Super lovely, but a lot of our products have problems as the nodata output defaults to zero. Eg. with NDVI, 0 means water not nodata. Explicitly stating the nodata default in the doco would help; additionally making it obvious that you can set the dtype would help. I know that rasterio arguments can be passed, but it took me about a month of using the function until I had the brains to look for this in the rasterio.open doco.....


Happy to help with this one. 


Discussion:
Type conversion is out of scope for this function, so do `ds.astype('float32')` on input instead.

`nodata=` argument is already accepted, also it is being read from attributes dictionary, but supplied argument takes precedence over `ds.nodata`. Although it is apparently missing from documentation.

For floating point images that contain `NaN`s there is no need to define `nodata` attribute at all, but you have to ensure that `ds.attrs["nodata"]` doesn't exist also, otherwise that value will be added to output file as `nodata` value.



Resolving patch:
diff --git a/datacube/utils/cog.py b/datacube/utils/cog.py
--- a/datacube/utils/cog.py
+++ b/datacube/utils/cog.py
@@ -202,6 +202,8 @@ def write_cog(geo_im: xr.DataArray,
     :param overview_resampling: Use this resampling when computing overviews
     :param overview_levels: List of shrink factors to compute overiews for: [2,4,8,16,32],
                             to disable overviews supply empty list ``[]``
+    :param nodata: Set ``nodata`` flag to this value if supplied, by default ``nodata`` is
+                   read from the attributes of the input array (``geo_im.attrs['nodata']``).
     :param extra_rio_opts: Any other option is passed to ``rasterio.open``
 
     :returns: Path to which output was written
@@ -271,6 +273,8 @@ def to_cog(geo_im: xr.DataArray,
     :param ovr_blocksize: Size of internal tiles in overview images (defaults to blocksize)
     :param overview_resampling: Use this resampling when computing overviews
     :param overview_levels: List of shrink factors to compute overiews for: [2,4,8,16,32]
+    :param nodata: Set ``nodata`` flag to this value if supplied, by default ``nodata`` is
+                   read from the attributes of the input array (``geo_im.attrs['nodata']``).
     :param extra_rio_opts: Any other option is passed to ``rasterio.open``
 
     :returns: In-memory GeoTiff file as bytes


Test patch:
Repository: opendatacube/datacube-core

Issue:
Example PickleWriterDriver does not support URIs
### Expected behaviour
Using the example `pickles` driver should allow to successfully run, e.g., `integration_tests/test_end_to_end`. This is assuming the example dcio driver has been installed first.

### Actual behaviour
The test fails because a URI gets passed to `PickleWriterDriver.write_dataset_to_storage()` which still expects a filepath string. Also, this URI may contain intermediate directories that need creating.

### Steps to reproduce the behaviour
- Set `pickles` as driver name in, e.g., `datacube-core/docs/config_samples/ingester/ls5_nbar_albers.yaml(106)`
- Run `pytest integration_tests/test_end_to_end.py`
- First error: `AttributeError: 'PickleWriterDriver' object has no attribute 'mk_uri'` but subsequent errors happen due to missing intermediate directories.

### Environment information
- `Open Data Cube core, version 1.8.4.dev52+g07bc51a5.d20210222`
- Docker image: `opendatacube/datacube-tests:latest`


Discussion:

Resolving patch:
diff --git a/examples/io_plugin/dcio_example/pickles.py b/examples/io_plugin/dcio_example/pickles.py
--- a/examples/io_plugin/dcio_example/pickles.py
+++ b/examples/io_plugin/dcio_example/pickles.py
@@ -4,9 +4,12 @@
 # SPDX-License-Identifier: Apache-2.0
 """ Example reader plugin
 """
-from contextlib import contextmanager
 import pickle
+from contextlib import contextmanager
+from pathlib import Path
+from urllib.parse import urlsplit
 
+from datacube.utils.uris import normalise_path
 
 PROTOCOL = 'file'
 FORMAT = 'pickle'
@@ -103,12 +106,33 @@ def format(self):
     def uri_scheme(self):
         return PROTOCOL
 
-    def write_dataset_to_storage(self, dataset, filename,
+    def mk_uri(self, file_path, storage_config):
+        """
+        Constructs a URI from the file_path and storage config.
+
+        A typical implementation should return f'{scheme}://{file_path}'
+
+        Example:
+            file_path = '/path/to/my_file.pickled'
+            storage_config = {'driver': 'pickles'}
+
+            mk_uri(file_path, storage_config) should return 'file:///path/to/my_file.pickled'
+
+        :param Path file_path: The file path of the file to be converted into a URI.
+        :param dict storage_config: The dict holding the storage config found in the ingest definition.
+        :return: file_path as a URI that the Driver understands.
+        :rtype: str
+        """
+        return normalise_path(file_path).as_uri()
+
+    def write_dataset_to_storage(self, dataset, file_uri,
                                  global_attributes=None,
                                  variable_params=None,
                                  storage_config=None,
                                  **kwargs):
-        with open(filename, 'wb') as f:
+        filepath = Path(urlsplit(file_uri).path)
+        filepath.parent.mkdir(parents=True, exist_ok=True)
+        with filepath.open('wb') as f:
             pickle.dump(dataset, f)
         return {}
 


Test patch:
Repository: opendatacube/datacube-core

Issue:
2 character strings are treated as Range object by Query class
### Expected behaviour

```python
from datacube.api.query import Query
assert Query(k='AB').search['k'] == 'AB'
```

### Actual behaviour
Instead `k='AB'` is turned into `Range('A', 'B')`

Error is here:
https://github.com/opendatacube/datacube-core/blob/176406516994516f169160103482cd3f42b7b954/datacube/api/query.py#L249


Discussion:

Resolving patch:
diff --git a/datacube/api/query.py b/datacube/api/query.py
--- a/datacube/api/query.py
+++ b/datacube/api/query.py
@@ -246,8 +246,14 @@ def _values_to_search(**kwargs):
         if key.lower() in ('time', 't'):
             search['time'] = _time_to_search_dims(value)
         elif key not in ['latitude', 'lat', 'y'] + ['longitude', 'lon', 'x']:
-            if isinstance(value, collections.abc.Sequence) and len(value) == 2:
+            # If it's not a string, but is a sequence of length 2, then it's a Range
+            if (
+                not isinstance(value, str)
+                and isinstance(value, collections.abc.Sequence)
+                and len(value) == 2
+            ):
                 search[key] = Range(*value)
+            # All other cases are default
             else:
                 search[key] = value
     return search


Test patch:
diff --git a/tests/api/test_query.py b/tests/api/test_query.py
--- a/tests/api/test_query.py
+++ b/tests/api/test_query.py
@@ -208,3 +208,8 @@ def test_dateline_query_building():
                 crs='EPSG:32660').search_terms['lon']
 
     assert lon.begin < 180 < lon.end
+
+
+def test_query_issue_1146():
+    q = Query(k='AB')
+    assert q.search['k'] == 'AB'
Repository: opendatacube/datacube-core

Issue:
 cannot import name 'clock_gettime' from 'time' (unknown location)
### Expected behaviour
Import time

### Actual behaviour

  File "C:\ProgramData\Anaconda3\lib\site-packages\datacube\drivers\postgres\_connections.py", line 20, in <module>
    from time import clock_gettime, CLOCK_REALTIME

ImportError: cannot import name 'clock_gettime' from 'time' (unknown location)

cannot import name 'clock_gettime' from 'time' (unknown location)


### Steps to reproduce the behaviour
  from ._connections import PostgresDb

### Environment information
Python 3.8.11
* Which ``datacube --version`` are you using?
Unknown, it's referenced in geocube on conda-forge
* What datacube deployment/enviornment are you running against?
n/a

Additional notes:
time.clock has been removed in Python 3.8 after being deprecated in Python 3.3. The recommendation is to change clock by either time.perf_counter or time.process_time.



Discussion:
@bartzwemmer looks like it's `1.8.4` version, in the future you can check version with either running `datacube --version` command after activating the environment, or using `pip list` or `conda list` command.

@SpacemanPaul @omad @jeremyh problem was introduced in this PR #1168, `time.clock_gettime` is a Linux only method and will not work on windows. Note that we do not run any tests on windows at all.




Resolving patch:
diff --git a/datacube/drivers/postgres/_connections.py b/datacube/drivers/postgres/_connections.py
--- a/datacube/drivers/postgres/_connections.py
+++ b/datacube/drivers/postgres/_connections.py
@@ -17,7 +17,6 @@
 import os
 import re
 from contextlib import contextmanager
-from time import clock_gettime, CLOCK_REALTIME
 from typing import Callable, Optional, Union
 
 from sqlalchemy import event, create_engine, text
@@ -272,6 +271,10 @@ def handle_dynamic_token_authentication(engine: Engine,
     @event.listens_for(engine, "do_connect")
     def override_new_connection(dialect, conn_rec, cargs, cparams):
         # Handle IAM authentication
+        # Importing here because the function `clock_gettime` is not available on Windows
+        # which shouldn't be a problem, because boto3 auth is mostly used on AWS.
+        from time import clock_gettime, CLOCK_REALTIME
+
         now = clock_gettime(CLOCK_REALTIME)
         if now - last_token_time[0] > timeout:
             last_token[0] = new_token(**kwargs)


Test patch:
Repository: opendatacube/datacube-core

Issue:
Shapely deprecation warning 
version: latest tagged
```
azonaws.com/opendatacube/explorer@sha256:1baee618d320969cc1a9c910ef33c5a9bfcb2b3e7872fde7c2d1c8fb34944f8eexplorer-prod-datacube-explorer-85bb977596-s6lh2 | {"log":"/usr/local/lib/python3.8/dist-packages/datacube/utils/geometry/_base.py:698: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. ","stream":"stderr"}
```


Discussion:
Thanks for the report. It looks like we need to push this up to datacube core, as the iteration is happening in its code: https://github.com/opendatacube/datacube-core/blob/843bd4e6c05e963f3b6d29bef54f974c99e06670/datacube/utils/geometry/_base.py#L697-L699


Resolving patch:
diff --git a/datacube/utils/geometry/_base.py b/datacube/utils/geometry/_base.py
--- a/datacube/utils/geometry/_base.py
+++ b/datacube/utils/geometry/_base.py
@@ -705,7 +705,8 @@ def split(self, splitter: 'Geometry') -> Iterable['Geometry']:
             yield Geometry(g, self.crs)
 
     def __iter__(self) -> Iterator['Geometry']:
-        for geom in self.geom:
+        sub_geoms = getattr(self.geom, "geoms", [])
+        for geom in sub_geoms:
             yield Geometry(geom, self.crs)
 
     def __nonzero__(self) -> bool:


Test patch:
Repository: opendatacube/datacube-core

Issue:
Don't use rasterio.path
@Kirill888 exposing rasterio.path and its members in Rasterio's public API was a mistake (https://github.com/rasterio/rasterio/issues/2365). Can you change https://github.com/opendatacube/datacube-core/blob/95e72fdbff2707aa6f7abc837d9eab52c50258e1/datacube/storage/_rio.py#L16 to not use it and to just call rasterio.open() instead?

I would love to rename rasterio.path to rasterio._path. At the very least we will warning about this in 1.3.0.


Discussion:
Thanks for the heads up @sgillies. At the time this was introduced, it was really just a work-around for some performance issues caused by `AWSSession` construction bugs that were long time since been fixed, I believe it was attempting to get credentials when no credentials settings were set or something like that.

@SpacemanPaul this look like it's a trivial fix: replace `DatasetReader(...)` with `rasterio.open(...)` and pass through uri as a string. So would be nice for it to be included in the next release.







Resolving patch:
diff --git a/datacube/storage/_rio.py b/datacube/storage/_rio.py
--- a/datacube/storage/_rio.py
+++ b/datacube/storage/_rio.py
@@ -13,7 +13,6 @@
 import numpy as np
 from affine import Affine
 import rasterio
-import rasterio.path
 from urllib.parse import urlparse
 from typing import Optional, Iterator
 
@@ -165,8 +164,7 @@ def open(self) -> Iterator[GeoRasterReader]:
 
         try:
             _LOG.debug("opening %s", self.filename)
-            with rasterio.DatasetReader(rasterio.path.parse_path(str(self.filename)),
-                                        sharing=False) as src:
+            with rasterio.open(str(self.filename), sharing=False) as src:
                 override = False
 
                 transform = src.transform


Test patch:
Repository: opendatacube/datacube-core

Issue:
Missing Jupyter Notebook documentation in docs
### Expected behaviour
Hi all, a while ago I updated the docs to include a better intro to querying products and datasets and loading data: https://github.com/opendatacube/datacube-core/pull/1244/files

From what I can tell, those new files are correctly included in the index:
https://github.com/opendatacube/datacube-core/blob/develop/docs/data-access-analysis/index.rst

### Actual behaviour
However, I can't seem to see these in the docs:
https://opendatacube.readthedocs.io/en/latest/data-access-analysis/apis/datacube-class.html

![image](https://user-images.githubusercontent.com/17680388/174537025-fd22bf17-1d3b-4e4f-ae80-4ba34a64916a.png)


Can anyone see anything obviously wrong? Do we need to do something special to get Jupyter Notebook docs to show up? (the only one that does show correctly is the .rst file in that list)


### Steps to reproduce the behaviour

Visit https://opendatacube.readthedocs.io/en/latest/data-access-analysis/apis/datacube-class.html

### Environment information

* N/A




Discussion:

Resolving patch:
diff --git a/docs/click_utils.py b/docs/click_utils.py
--- a/docs/click_utils.py
+++ b/docs/click_utils.py
@@ -77,3 +77,8 @@ def setup(app):
     app.add_config_value('click_utils_commands', {}, 'html')
 
     app.add_domain(DatacubeDomain)
+    return {
+        'parallel_read_safe': False,
+        'parallel_write_safe': False,
+    }
+
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -35,6 +35,7 @@
     'sphinx_click.ext',
     'click_utils',
     'autodocsumm',
+    'nbsphinx',
     'sphinx.ext.napoleon'
 ]
 
@@ -68,7 +69,7 @@
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-exclude_patterns = ['README.rst']
+exclude_patterns = ['README.rst', '.condaenv', '.direnv']
 
 # If true, '()' will be appended to :func: etc. cross-reference text.
 add_function_parentheses = True


Test patch:
Repository: opendatacube/datacube-core

Issue:
Code includes Type Annotations,  but they're not made available for type checking (PEP561)
**Summary**

The ODC code is fairly well annotated with [Python type hints](https://typing.readthedocs.io/en/latest/), but these type hints aren't made availble for use in downstream packages by following [PEP 561 – Distributing and Packaging Type Information | peps.python.org](https://peps.python.org/pep-0561/).

**Proposal**

Since ODC Core includes inline type hints with the code, we need to follow [packaging type information - PEP561](https://peps.python.org/pep-0561/#packaging-type-information) by adding an empty `datacube/py.typed` file, and ensuring it's distributed by adding it to `package_data` in `setup.py`.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -82,6 +82,7 @@
     ),
     package_data={
         '': ['*.yaml', '*/*.yaml'],
+        'datacube': ['py.typed'],
     },
     scripts=[],
     install_requires=[


Test patch:
Repository: opendatacube/datacube-core

Issue:
[doc] missing information on group_by
lacking information on `group_by` in the doc


Discussion:

Resolving patch:
diff --git a/datacube/api/core.py b/datacube/api/core.py
--- a/datacube/api/core.py
+++ b/datacube/api/core.py
@@ -262,6 +262,7 @@ def load(self, product=None, measurements=None, output_crs=None, resolution=None
             a function can be provided to the merging into a single time slice.
 
             See :func:`datacube.helpers.ga_pq_fuser` for an example implementation.
+            see :func:`datacube.api.query.query_group_by` for `group_by` built-in functions.
 
 
         **Output**
diff --git a/datacube/api/query.py b/datacube/api/query.py
--- a/datacube/api/query.py
+++ b/datacube/api/query.py
@@ -29,6 +29,8 @@
 class GroupBy:
     def __init__(self, group_by_func, dimension, units, sort_key=None, group_key=None):
         """
+        GroupBy Object
+
         :param group_by_func: Dataset -> group identifier
         :param dimension: dimension of the group key
         :param units: units of the group key
@@ -190,9 +192,28 @@ def _extract_time_from_ds(ds: Dataset) -> datetime.datetime:
 
 
 def query_group_by(group_by='time', **kwargs):
-    if not isinstance(group_by, str):
+    """
+    Group by function for loading datasets
+
+    :param group_by: group_by name, supported str are
+    ::
+
+        - time (default)
+        - solar_day, see :func:`datacube.api.query.solar_day`
+
+    or ::
+
+        - :class:`datacube.api.query.GroupBy` object
+
+    :return: :class:`datacube.api.query.GroupBy`
+    :raises LookupError: when group_by string is not a valid dictionary key.
+    """
+    if isinstance(group_by, GroupBy):
         return group_by
 
+    if not isinstance(group_by, str):
+        group_by = None
+
     time_grouper = GroupBy(group_by_func=_extract_time_from_ds,
                            dimension='time',
                            units='seconds since 1970-01-01 00:00:00')
@@ -212,7 +233,9 @@ def query_group_by(group_by='time', **kwargs):
     try:
         return group_by_map[group_by]
     except KeyError:
-        raise LookupError('No group by function for', group_by)
+        raise LookupError(
+            f'No group by function for {group_by}, valid options are: {group_by_map.keys()}',
+        )
 
 
 def _range_to_geopolygon(**kwargs):


Test patch: