internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: opendatacube/datacube-core

Issue:
Incompatibilities with xarray > 2022.03
### Expected behaviour
ODC should work with current version of `xarray`. In `setup.py` there's an exclusion of `2022.6.0`, but I don't think that's sufficient. It'd be worth digging up the commit/PR that made that change.

### Actual behaviour

Tests are failing.

```
FAILED tests/api/test_grid_workflow.py::test_gridworkflow_with_time_depth - AssertionError
FAILED tests/api/test_virtual.py::test_aggregate - ValueError: time already exists as coordinate or variable name.
```

### Steps to reproduce the behaviour

`pytest tests/`

### Environment information

* Which ``datacube --version`` are you using?

 `develop` branch at `af59377327c363b9c52b55000b4024a0b3fbaa8b`

* What datacube deployment/enviornment are you running against?

  - Mambaforge
  - conda-forge 
  - Python 3.10






Discussion:
Seems like this is an issue with xarray, as raised by Emma: https://github.com/pydata/xarray/issues/6836

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -106,7 +106,7 @@
         'sqlalchemy',
         'GeoAlchemy2',
         'toolz',
-        'xarray>=0.9,!=2022.6.0',  # >0.9 fixes most problems with `crs` attributes being lost
+        'xarray>=0.9,<2022.6',  # >0.9 fixes most problems with `crs` attributes being lost
     ],
     extras_require=extras_require,
     tests_require=tests_require,


Test patch:
Repository: opendatacube/datacube-core

Issue:
Fix Deprecation warning against distutils
I've started seeing this Deprecation Warning, even when running `pytest` of all things!

```
warning : /home/omad/miniconda3/envs/odc-tools/lib/python3.9/site-packages/datacube/utils/geometry/_base.py:153: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  DEFAULT_WKT_VERSION = (WktVersion.WKT1_GDAL if LooseVersion(rasterio.__gdal_version__) < LooseVersion("3.0.0")
```

Datacube Core Version: 1.8.8


Discussion:
The fix required is to replace usages of `distutils.LooseVersion` with `packaging.version.Version` and adding `packaging` as a new dependency.

See:
- https://packaging.pypa.io/en/latest/version.html
- https://peps.python.org/pep-0632/

Resolving patch:
diff --git a/datacube/utils/geometry/_base.py b/datacube/utils/geometry/_base.py
--- a/datacube/utils/geometry/_base.py
+++ b/datacube/utils/geometry/_base.py
@@ -10,7 +10,7 @@
 from collections import namedtuple, OrderedDict
 from typing import Tuple, Iterable, List, Union, Optional, Any, Callable, Hashable, Dict, Iterator
 from collections.abc import Sequence
-from distutils.version import LooseVersion
+from packaging.version import Version
 
 import cachetools
 import numpy
@@ -150,7 +150,7 @@ class CRS:
     """
     Wrapper around `pyproj.CRS` for backwards compatibility.
     """
-    DEFAULT_WKT_VERSION = (WktVersion.WKT1_GDAL if LooseVersion(rasterio.__gdal_version__) < LooseVersion("3.0.0")
+    DEFAULT_WKT_VERSION = (WktVersion.WKT1_GDAL if Version(rasterio.__gdal_version__) < Version("3.0.0")
                            else WktVersion.WKT2_2019)
 
     __slots__ = ('_crs', '_epsg', '_str')
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -107,6 +107,7 @@
         'GeoAlchemy2',
         'toolz',
         'xarray>=0.9,<2022.6',  # >0.9 fixes most problems with `crs` attributes being lost
+        'packaging',
     ],
     extras_require=extras_require,
     tests_require=tests_require,


Test patch:
diff --git a/integration_tests/test_cli_output.py b/integration_tests/test_cli_output.py
--- a/integration_tests/test_cli_output.py
+++ b/integration_tests/test_cli_output.py
@@ -3,6 +3,7 @@
 # Copyright (c) 2015-2022 ODC Contributors
 # SPDX-License-Identifier: Apache-2.0
 
+
 def test_cli_product_subcommand(index_empty, clirunner, dataset_add_configs):
     runner = clirunner(['product', 'update'], verbose_flag=False, expect_success=False)
     assert "Usage:  [OPTIONS] [FILES]" in runner.output
Repository: opendatacube/datacube-core

Issue:
Online API documentation has gone missing
### Expected behaviour
I should be able to find API documentation online.

### Actual behaviour
![image](https://user-images.githubusercontent.com/108979/217148967-1aa771cc-ec4e-4719-bcd9-3fc307ff71f9.png)

### More info

The latest release of Datacube both added some new dependencies, and bumped the minimum version of existing ones. This has caused the [Read the Docs builds](https://readthedocs.org/projects/datacube-core/builds/19401442/) to start quietly failing.

![image](https://user-images.githubusercontent.com/108979/217149219-23e19540-a01b-4271-a186-94654b20cd89.png)



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -14,12 +14,13 @@
 ]
 doc_require = [
     'Sphinx',
-    'sphinx_rtd_theme',
     'sphinx_autodoc_typehints',  # Propagate mypy info into docs
     'sphinx-click',
     'recommonmark',
-    'setuptools',  # version related dependencies
-    'setuptools_scm[toml]',
+    'autodocsumm',
+    'beautifulsoup4',
+    'nbsphinx',
+    'pydata-sphinx-theme==0.9.0',
 ]
 
 extras_require = {


Test patch:
Repository: opendatacube/datacube-core

Issue:
links on read the doc footer leads to 404
https://github.com/opendatacube/datacube-core/blob/develop/docs/_templates/odc-footer.html renders at the footer of page https://datacube-core.readthedocs.io/en/latest/index.html


Discussion:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -70,7 +70,7 @@
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-exclude_patterns = ['README.rst', '.condaenv', '.direnv']
+exclude_patterns = ['README.rst', '.condaenv', '.direnv', '_build']
 
 # If true, '()' will be appended to :func: etc. cross-reference text.
 add_function_parentheses = True


Test patch:
Repository: opendatacube/datacube-core

Issue:
python 3.8/3.9 doesn't support entry_points with either group or select
As title.

https://github.com/opendatacube/datacube-core/blob/5fed4a688c85c198c5d25f2570b0c92ef91dfc04/datacube/drivers/driver_cache.py#L50

Ref: https://docs.python.org/3/library/importlib.metadata.html
>Compatibility Note: The “selectable” entry points were introduced in importlib_metadata 3.6 and Python 3.10. Prior to those changes, entry_points accepted no parameters and always returned a dictionary of entry points, keyed by group. For compatibility, if no parameters are passed to entry_points, a SelectableGroups object is returned, implementing that dict interface. In the future, calling entry_points with no parameters will return an EntryPoints object. Users should rely on the selection interface to retrieve entry points by group.


Discussion:

Resolving patch:
diff --git a/datacube/drivers/driver_cache.py b/datacube/drivers/driver_cache.py
--- a/datacube/drivers/driver_cache.py
+++ b/datacube/drivers/driver_cache.py
@@ -46,7 +46,10 @@ def safe_load(ep):
         return driver
 
     def resolve_all(group: str) -> Iterable[Tuple[str, Any]]:
-        from importlib.metadata import entry_points
+        try:
+            from importlib_metadata import entry_points
+        except ModuleNotFoundError:
+            from importlib.metadata import entry_points
         for ep in entry_points(group=group):
             driver = safe_load(ep)
             if driver is not None:
diff --git a/docs/click_utils.py b/docs/click_utils.py
--- a/docs/click_utils.py
+++ b/docs/click_utils.py
@@ -2,7 +2,6 @@
 #
 # Copyright (c) 2015-2023 ODC Contributors
 # SPDX-License-Identifier: Apache-2.0
-from importlib.metadata import entry_points
 from docutils.nodes import literal_block, section, title, make_id
 from sphinx.domains import Domain
 from docutils.parsers.rst import Directive
@@ -34,6 +33,10 @@ def find_script_callable_from_env(name, env):
 
 
 def find_script_callable(name):
+    try:
+        from importlib_metadata import entry_points
+    except ModuleNotFoundError:
+        from importlib.metadata import entry_points
     return list(entry_points(
         group='console_scripts', name=name))[0].load()
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -112,6 +112,7 @@
         'xarray>=0.9',  # >0.9 fixes most problems with `crs` attributes being lost
         'packaging',
         'deprecat',
+        'importlib_metadata>3.5;python_version<"3.10"',
     ],
     extras_require=extras_require,
     tests_require=tests_require,


Test patch:
Repository: opendatacube/datacube-core

Issue:
list_products ignores information in load hints
Products can specify preferred CRS and resolution for data loading via "load hints". Those are ignored when generating product summary table with `Datacube.list_products()` method.

https://github.com/opendatacube/datacube-core/blob/47f0aca621fd2334298181f0647febc337943f49/datacube/api/core.py#L124-L133

Code above only looks at `grid_spec`, which won't be defined for product with load hints only.




Discussion:

Resolving patch:
diff --git a/datacube/api/core.py b/datacube/api/core.py
--- a/datacube/api/core.py
+++ b/datacube/api/core.py
@@ -113,6 +113,14 @@ def list_products(self, with_pandas=True, dataset_count=False):
         :return: A table or list of every product in the datacube.
         :rtype: pandas.DataFrame or list(dict)
         """
+        def _get_non_default(product, col):
+            load_hints = product.load_hints()
+            if load_hints:
+                if col == 'crs':
+                    return load_hints.get('output_crs', None)
+                return load_hints.get(col, None)
+            return getattr(product.grid_spec, col, None)
+
         # Read properties from each datacube product
         cols = [
             'name',
@@ -125,10 +133,10 @@ def list_products(self, with_pandas=True, dataset_count=False):
             getattr(pr, col, None)
             # if 'default_crs' and 'default_resolution' are not None
             # return 'default_crs' and 'default_resolution'
-            if getattr(pr, col, None) and 'default' not in col
-            # else try 'grid_spec.crs' and 'grid_spec.resolution'
+            if getattr(pr, col, None) or 'default' not in col
+            # else get crs and resolution from load_hints or grid_spec
             # as per output_geobox() handling logic
-            else getattr(pr.grid_spec, col.replace('default_', ''), None)
+            else _get_non_default(pr, col.replace('default_', ''))
             for col in cols]
             for pr in self.index.products.get_all()]
 


Test patch:
Repository: opendatacube/datacube-core

Issue:
Empty `~/.datacube.conf` causes errors
### Expected behaviour

```
mv ~/.datacube.conf datacube.conf.bk
touch ~/.datacube.conf

datacube system check
```


### Actual behaviour

`datacube system check` fails with traceback.

in the notebook, traceback is more clear, `self.raw_config is None` but is being iterated over. 

```python
File ~/.envs/emit/lib/python3.10/site-packages/datacube/cfg/api.py:121, in ODCConfig.__init__(self, paths, raw_dict, text)
    118     self.raw_config = parse_text(cast(str, self.raw_text))
    120 self._aliases: dict[str, str] = {}
--> 121 self.known_environments: dict[str, ODCEnvironment] = {
    122     section: ODCEnvironment(self, section, self.raw_config[section], self.allow_envvar_overrides)
    123     for section in self.raw_config
    124 }
    125 self.canonical_names: dict[str, list[str]] = {}
    126 for alias, canonical in self._aliases.items():

TypeError: 'NoneType' object is not iterable
```

### Steps to reproduce the behaviour

that's on `develop-1.9` branch, just create `.datacube.conf` file without any data in it.


Discussion:
Ah, yes.  A blank config should parse as an empty dict `{}` - good catch.

Resolving patch:
diff --git a/datacube/cfg/api.py b/datacube/cfg/api.py
--- a/datacube/cfg/api.py
+++ b/datacube/cfg/api.py
@@ -104,7 +104,7 @@ def __init__(
         self.allow_envvar_overrides = not text and not raw_dict
 
         if not raw_dict and not text:
-            # No explict config passed in.  Check for ODC_CONFIG environmnet variables
+            # No explict config passed in.  Check for ODC_CONFIG environment variables
             if os.environ.get("ODC_CONFIG"):
                 text = os.environ["ODC_CONFIG"]
             else:
diff --git a/datacube/cfg/utils.py b/datacube/cfg/utils.py
--- a/datacube/cfg/utils.py
+++ b/datacube/cfg/utils.py
@@ -54,5 +54,5 @@ def smells_like_ini(cfg_text: str):
             return True
         else:
             return False
-    # Doesn't smell like anything
-    return False
+    # Empty file - parse as ini so we end up with the right high level structure at least
+    return True


Test patch:
diff --git a/tests/test_cfg.py b/tests/test_cfg.py
--- a/tests/test_cfg.py
+++ b/tests/test_cfg.py
@@ -45,7 +45,7 @@ def test_smells_like_ini():
 aasdfer\\faw043[]]][""")
 
     # Pure white space
-    assert not smells_like_ini("   \n  \n    \n \n  \t  \t  \n   \n")
+    assert smells_like_ini("   \n  \n    \n \n  \t  \t  \n   \n")
 
 
 @pytest.fixture
Repository: AlexsLemonade/refinebio

Issue:
Expose transformation option to API
### Context

https://github.com/AlexsLemonade/refinebio-frontend/issues/208

### Problem or idea

We want a dropdown to change the transformation option, but the API currently doesn't support changing that value.

### Solution or next step

I think transformation just needs to be added to the DataSetSerializer

### New Issue Checklist

- [x] The title is short and descriptive.
- [x] You have explained the context that led you to write this issue.
- [x] You have reported a problem or idea.
- [x] You have proposed a solution or next step.



Discussion:

Resolving patch:
diff --git a/api/data_refinery_api/serializers.py b/api/data_refinery_api/serializers.py
--- a/api/data_refinery_api/serializers.py
+++ b/api/data_refinery_api/serializers.py
@@ -416,6 +416,7 @@ class Meta:
                     'id',
                     'data',
                     'aggregate_by',
+                    'scale_by',
                     'is_processing',
                     'is_processed',
                     'is_available',


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Default sorting for `/samples` endpoint
### Context

From https://github.com/AlexsLemonade/refinebio-frontend/issues/365#issuecomment-429107314 

> In addition: In experiments where only some of the samples have been displayed, show the addable samples first.

### Problem or idea

By default, we need to show the processed samples first.

### Solution or next step

Modify the default sorting of the `/samples` endpoint, so that the processed samples are returned first.

### New Issue Checklist

- [x] The title is short and descriptive.
- [x] You have explained the context that led you to write this issue.
- [x] You have reported a problem or idea.
- [x] You have proposed a solution or next step.



Discussion:

Resolving patch:
diff --git a/api/data_refinery_api/views.py b/api/data_refinery_api/views.py
--- a/api/data_refinery_api/views.py
+++ b/api/data_refinery_api/views.py
@@ -563,7 +563,7 @@ def get(self, request, format=None):
             # Python doesn't provide a prettier way of doing this that I know about.
             filter_dict['accession_code__in'] = [item for sublist in dataset.data.values() for item in sublist]
 
-        samples = Sample.public_objects.filter(**filter_dict)
+        samples = Sample.public_objects.filter(**filter_dict).order_by('-is_processed')
         if order_by:
             samples = samples.order_by(order_by)
 


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Un-hard-code ensembl version
### Context

https://github.com/AlexsLemonade/refinebio/pull/1308

### Problem or idea

Ensembl broke their version endpoint so we hardcoded the correct version. They've since fixed their version endpoint, so we can go back to pulling the version from there instead of hard coding it.

### Solution or next step

Use ensembl's API to pull the latest version rather than always using version 43.


Discussion:

Resolving patch:
diff --git a/foreman/data_refinery_foreman/surveyor/transcriptome_index.py b/foreman/data_refinery_foreman/surveyor/transcriptome_index.py
--- a/foreman/data_refinery_foreman/surveyor/transcriptome_index.py
+++ b/foreman/data_refinery_foreman/surveyor/transcriptome_index.py
@@ -60,12 +60,7 @@ def __init__(self, species: Dict):
         self.url_root = "ensemblgenomes.org/pub/release-{assembly_version}/{short_division}"
         self.short_division = DIVISION_LOOKUP[species["division"]]
         self.assembly = species["assembly_name"].replace(" ", "_")
-
-        # For some reason the API is returning version 44, which doesn't seem to exist
-        # in the FTP servers: ftp://ftp.ensemblgenomes.org/pub/
-        # That's why the version is hardcoded below.
-        # self.assembly_version = utils.requests_retry_session().get(DIVISION_RELEASE_URL).json()["version"]
-        self.assembly_version = '43'
+        self.assembly_version = utils.requests_retry_session().get(DIVISION_RELEASE_URL).json()["version"]
 
         self.species_sub_dir = species["name"]
         self.filename_species = species["name"].capitalize()


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Email: Link 'Download' to dataset page instead of S3
### Context
Loosely related to https://github.com/AlexsLemonade/refinebio-frontend/issues/633#issuecomment-491928496

### Problem or idea

When the user receives the email with the dataset, the 'Download Now' button links to the S3 bucket. If the dataset is expired, the user sees this message: 
![Screen Shot 2019-05-14 at 9 19 53 AM](https://user-images.githubusercontent.com/15315514/57701204-7fb7cf00-7629-11e9-8122-aa7c1783868f.png)

They cannot regenerate it, even though that's an option we provide. 

### Solution or next step

Change the CTA to 'Get Dataset' and link it to the dataset page instead. 

![Screen Shot 2019-05-14 at 9 28 55 AM](https://user-images.githubusercontent.com/15315514/57701789-b7734680-762a-11e9-8e86-feaddaeeaa47.png)

We can also unlink 'dataset' in `Your dataset is ready for download` but it is not crucial. 



Discussion:

Resolving patch:
diff --git a/workers/data_refinery_workers/processors/smasher.py b/workers/data_refinery_workers/processors/smasher.py
--- a/workers/data_refinery_workers/processors/smasher.py
+++ b/workers/data_refinery_workers/processors/smasher.py
@@ -934,8 +934,8 @@ def _notify(job_context: Dict) -> Dict:
                 job_context['success'] = False
             else:
                 SUBJECT = "Your refine.bio Dataset is Ready!"
-                BODY_TEXT = "Hot off the presses:\n\n" + job_context["result_url"] + "\n\nLove!,\nThe refine.bio Team"
-                FORMATTED_HTML = BODY_HTML.replace('REPLACE_DOWNLOAD_URL', job_context["result_url"])\
+                BODY_TEXT = "Hot off the presses:\n\n" + dataset_url + "\n\nLove!,\nThe refine.bio Team"
+                FORMATTED_HTML = BODY_HTML.replace('REPLACE_DOWNLOAD_URL', dataset_url)\
                                           .replace('REPLACE_DATASET_URL', dataset_url)
 
             # Try to send the email.


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
GSE56035: refinebio_age isn't quite right
### Context
The harmonization on some metadata isn't correct due to the weirdness of how the metadata was written.

### Problem or idea
For GSE56035, there are two columns `age` and `age (yrs)`. refinebio_age is populated only with the values in `age` so it looks like there is a lot of missing data when that exists in another column `age (yrs)` 

### Solution or next step

Tracking this here so that when we use the MetaSRA annotations we can check if this is fixed. 


Discussion:

Resolving patch:
diff --git a/foreman/data_refinery_foreman/surveyor/harmony.py b/foreman/data_refinery_foreman/surveyor/harmony.py
--- a/foreman/data_refinery_foreman/surveyor/harmony.py
+++ b/foreman/data_refinery_foreman/surveyor/harmony.py
@@ -349,6 +349,7 @@ def harmonize(metadata: List) -> Dict:
                     'patient age',
                     'age of patient',
                     'age (years)',
+                    'age (yrs)',
                     'age (months)',
                     'age (days)',
                     'age (hours)',


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Dashboard does not accurately reflect job counts
### Context

We were checking the dashboard this morning to assess the health of the processing, but the dashboard figures were wrong.

### Problem or idea

We know from the database that we had one successful survey job finish between 5 and 6 PM, but it doesn't show up on the graph:
![image](https://user-images.githubusercontent.com/13942258/62711816-b2bef400-b9c7-11e9-8dad-c2e9ac47636f.png)

We also know that we've had tons of successful downloader jobs in the last 12 hours, but those also aren't on the graph:
![image](https://user-images.githubusercontent.com/13942258/62711952-e7cb4680-b9c7-11e9-8b43-000eb5344408.png)


### Solution or next step

We should look at our dashboard queries and see how they are different from the ones @kurtwheeler uses to check up on our service.



Discussion:

Resolving patch:
diff --git a/api/data_refinery_api/views.py b/api/data_refinery_api/views.py
--- a/api/data_refinery_api/views.py
+++ b/api/data_refinery_api/views.py
@@ -1043,7 +1043,7 @@ def _get_object_stats(cls, objects, range_param=False, field='created_at'):
         return result
 
     @classmethod
-    def _get_intervals(cls, objects, range_param, field='created_at'):
+    def _get_intervals(cls, objects, range_param, field='last_modified'):
         range_to_trunc = {
             'day': 'hour',
             'week': 'day',
@@ -1058,7 +1058,7 @@ def _get_intervals(cls, objects, range_param, field='created_at'):
             'year': current_date - timedelta(days=365)
         }
 
-        # trucate the `created_at` field by hour, day or month depending on the `range` param
+        # truncate the `last_modified` field by hour, day or month depending on the `range` param
         # and annotate each object with that. This will allow us to count the number of objects
         # on each interval with a single query
         # ref https://stackoverflow.com/a/38359913/763705


Test patch: