Repository: opendatacube/datacube-core

Issue:
New feature: load hints in product definition
Currently when using `dc.load(..)` on a non-ingested product user must specify output `CRS` and `resolution` to use for loading data. This has been raised numerous times as a number one usability concern for non-ingested products.

I believe an easy solution to this problem is to specify an "appropriate projection and resolution" to use when loading data at the product definition time. For example for Australian installation that uses Landsat 8, one could define default projection to be Australian Albers and default resolution to be 30m.

```yaml
load:
  crs: 'epsg:3577'
  resolution: 
    y: -30
    x: 30
  align:
    y: 0
    x: 0
```

To fully define pixel grid one also needs to "anchor" grid lines, default anchoring is to have pixel boundaries go through `0,0`, this is where `load.align.{x,y}` comes in. Code snippet above translates to something like

*When loading data for this product default to Australian Albers projection, use 30m pixels with inverted y-axis. Align pixel grid such that lines `x=0m` and `y=0m` fall on the pixel edge.*

With this data defined on a product, `dc.load` would use data specified in the product definition only if user hasn't supplied alternative configuration. Behavior of `dc.load` will remain the same for products that do not define `load` section in their metadata.

To be consistent with ingestor and `storage` configurations one would need to use `longitude,latitude` keys when default CRS is defined to be `epsg:4326`.





Discussion:
My first thought is that this sounds really great, and will hopefully ease the user experience a lot.

The only draw back is that the naive user will now be reverting back to having their data reprojected by default (e.g. just like back when all data was ingested vs. indexed). That means that unless they are careful, their data is going to be affected by some of the resampling issues that come with that (the only nice thing about the annoying UX flaw was it forced users to make careful decisions about resampling, which is something that often goes under the radar).

Not saying I disagree with the new feature proposal (I think it's great), just that it might require some careful explanation of what is happening behind the scenes so users can make an informed decision about how they load their data in the future.
I think this is a great idea, as I have observed a lot of confusion around projections and pixel alignments among users with less of a background in remote sensing.  I agree with @robbibt that if this change takes place we should make a concerted effort to document the default settings (and their downstream impacts) in the relevant notebooks/docs.  
Agree with @robbibt, ideally this feature would be complemented with `dc.load(.., resolution='native')` or similar. Problem with native load is "what to do when querying across UTM boundaries" for products like Landsat and Sentinel, do you fail or do you pick some kind of "majority". And then of course datacube can contain "anything", typically resolution is fixed per product-band pair, but there is no guarantee that it actually is, and a completely generic solution is kinda hard. Even picking majority is not that straightforward, is it just majority of datasets? Cause ideally it should be "majority of clear pixels", but that's much harder, and what if some pixels are different size? Looks like one would have to define a number of "policies" things like, "error if you can't use native projection for all the data", "use majority to pick projection", "use highest/lowest resolution across all bands/datasets being loaded".

Maybe this should also be augmented with per-band resampling configuration as was discussed here #779.
While implementing this feature I have stumbled into a realization that `storage` section has been coerced into the role I'm proposing for load hints, albeit without sufficient documentation.

https://github.com/opendatacube/datacube-core/blob/53374cc823f79cc720249a7a709aef686e4a69b5/datacube/api/core.py#L660-L668

Looks like partially defined `GridSpec` was used to communicate "desired CRS, and optionally resolution" to `dc.load(..)`. This behaviour has been "fixed" by #1012, from that change onward product will not report GridSpec if it's not fully defined, and so if people relied on this undocumented feature it will not work anymore, unless we repurpose `storage` to mean either

1. Fully specified gridding regime of an ingested product
2. Load hints, possibly partially specified (omitting resolution is allowed by current implementation by the looks of it)

I feel that having such an overloaded meaning for `storage` component of product definition is asking for trouble and might hinder future optimizations for ingested products. Also creating `GridSpec` class with missing components just to pass on information from product definition to loading code is just wrong. As a recipient of GridSpec object I should not need to check if it's fully defined.


Resolving patch:
diff --git a/datacube/api/core.py b/datacube/api/core.py
--- a/datacube/api/core.py
+++ b/datacube/api/core.py
@@ -301,6 +301,7 @@ def load(self, product=None, measurements=None, output_crs=None, resolution=None
 
         geobox = output_geobox(like=like, output_crs=output_crs, resolution=resolution, align=align,
                                grid_spec=datacube_product.grid_spec,
+                               load_hints=datacube_product.load_hints(),
                                datasets=datasets, **query)
 
         group_by = query_group_by(**query)
@@ -639,7 +640,7 @@ def with_fuser(m, fuser, default=None):
 
 
 def output_geobox(like=None, output_crs=None, resolution=None, align=None,
-                  grid_spec=None, datasets=None, geopolygon=None, **query):
+                  grid_spec=None, load_hints=None, datasets=None, geopolygon=None, **query):
     """ Configure output geobox from user provided output specs. """
 
     if like is not None:
@@ -651,26 +652,41 @@ def output_geobox(like=None, output_crs=None, resolution=None, align=None,
 
         return like.geobox
 
+    if load_hints:
+        if output_crs is None:
+            output_crs = load_hints.get('output_crs', None)
+
+        if resolution is None:
+            resolution = load_hints.get('resolution', None)
+
+        if align is None:
+            align = load_hints.get('align', None)
+
     if output_crs is not None:
-        # user provided specifications
         if resolution is None:
             raise ValueError("Must specify 'resolution' when specifying 'output_crs'")
         crs = geometry.CRS(output_crs)
-    else:
+    elif grid_spec is not None:
         # specification from grid_spec
-        if grid_spec is None or grid_spec.crs is None:
-            raise ValueError("Product has no default CRS. Must specify 'output_crs' and 'resolution'")
         crs = grid_spec.crs
         if resolution is None:
-            if grid_spec.resolution is None:
-                raise ValueError("Product has no default resolution. Must specify 'resolution'")
             resolution = grid_spec.resolution
         align = align or grid_spec.alignment
+    else:
+        raise ValueError("Product has no default CRS. Must specify 'output_crs' and 'resolution'")
 
+    # Try figuring out bounds
+    #  1. Explicitly defined with geopolygon
+    #  2. Extracted from x=,y=
+    #  3. Computed from dataset footprints
+    #  4. fail with ValueError
     if geopolygon is None:
         geopolygon = query_geopolygon(**query)
 
         if geopolygon is None:
+            if datasets is None:
+                raise ValueError("Bounds are not specified")
+
             geopolygon = get_bounds(datasets, crs)
 
     return geometry.GeoBox.from_geopolygon(geopolygon, resolution, crs, align)
diff --git a/datacube/model/__init__.py b/datacube/model/__init__.py
--- a/datacube/model/__init__.py
+++ b/datacube/model/__init__.py
@@ -404,6 +404,7 @@ def __init__(self,
         self.definition = definition
         self._canonical_measurements: Optional[Mapping[str, Measurement]] = None
         self._all_measurements: Optional[Dict[str, Measurement]] = None
+        self._load_hints: Optional[Dict[str, Any]] = None
 
     def _resolve_aliases(self):
         if self._all_measurements is not None:
@@ -514,6 +515,71 @@ def lookup_measurements(self, measurements: Optional[List[str]] = None) -> Mappi
         mm = self._resolve_aliases()
         return OrderedDict((m, mm[m]) for m in measurements)
 
+    def _extract_load_hints(self) -> Optional[Dict[str, Any]]:
+        _load = self.definition.get('load')
+        if _load is None:
+            # Check for partial "storage" definition
+            storage = self.definition.get('storage', {})
+
+            if 'crs' in storage and 'resolution' in storage:
+                if 'tile_size' in storage:
+                    # Fully defined GridSpec, ignore it
+                    return None
+
+                # TODO: warn user to use `load:` instead of `storage:`??
+                _load = storage
+            else:
+                return None
+
+        crs = geometry.CRS(_load['crs'])
+
+        def extract_point(name):
+            xx = _load.get(name, None)
+            return None if xx is None else tuple(xx[dim] for dim in crs.dimensions)
+
+        params = {name: extract_point(name) for name in ('resolution', 'align')}
+        params = {name: v for name, v in params.items() if v is not None}
+        return dict(crs=crs, **params)
+
+    @property
+    def default_crs(self) -> Optional[geometry.CRS]:
+        return self.load_hints().get('output_crs', None)
+
+    @property
+    def default_resolution(self) -> Optional[Tuple[float, float]]:
+        return self.load_hints().get('resolution', None)
+
+    @property
+    def default_align(self) -> Optional[Tuple[float, float]]:
+        return self.load_hints().get('align', None)
+
+    def load_hints(self) -> Dict[str, Any]:
+        """
+        Returns dictionary with keys compatible with ``dc.load(..)`` named arguments:
+
+          output_crs - CRS
+          resolution - Tuple[float, float]
+          align      - Tuple[float, float] (if defined)
+
+        Returns {} if load hints are not defined on this product, or defined with errors.
+        """
+        if self._load_hints is not None:
+            return self._load_hints
+
+        hints = None
+        try:
+            hints = self._extract_load_hints()
+        except Exception:
+            pass
+
+        if hints is None:
+            self._load_hints = {}
+        else:
+            crs = hints.pop('crs')
+            self._load_hints = dict(output_crs=crs, **hints)
+
+        return self._load_hints
+
     def dataset_reader(self, dataset_doc):
         return self.metadata_type.dataset_reader(dataset_doc)
 
diff --git a/datacube/testutils/__init__.py b/datacube/testutils/__init__.py
--- a/datacube/testutils/__init__.py
+++ b/datacube/testutils/__init__.py
@@ -157,7 +157,8 @@ def mk_sample_product(name,
                       measurements=('red', 'green', 'blue'),
                       with_grid_spec=False,
                       metadata_type=None,
-                      storage=None):
+                      storage=None,
+                      load=None):
 
     if storage is None and with_grid_spec is True:
         storage = {'crs': 'EPSG:3577',
@@ -200,6 +201,9 @@ def mk_measurement(m):
     if storage is not None:
         definition['storage'] = storage
 
+    if load is not None:
+        definition['load'] = load
+
     return DatasetType(metadata_type, definition)
 
 


Test patch:
diff --git a/tests/test_model.py b/tests/test_model.py
--- a/tests/test_model.py
+++ b/tests/test_model.py
@@ -1,10 +1,14 @@
 # coding=utf-8
 import pytest
 import numpy
+from copy import deepcopy
 from datacube.testutils import mk_sample_dataset, mk_sample_product
-from datacube.model import GridSpec, Measurement, MetadataType
+from datacube.model import GridSpec, Measurement, MetadataType, DatasetType
 from datacube.utils import geometry
+from datacube.utils.documents import InvalidDocException
 from datacube.storage import measurement_paths
+from datacube.testutils.geom import AlbersGS
+from datacube.api.core import output_geobox
 
 
 def test_gridspec():
@@ -129,6 +133,83 @@ def test_product_scale_factor():
     assert attrs['add_offset'] == -5
 
 
+def test_product_load_hints():
+    product = mk_sample_product('test_product',
+                                load=dict(crs='epsg:3857',
+                                          resolution={'x': 10, 'y': -10}))
+
+    assert 'load' in product.definition
+    assert DatasetType.validate(product.definition) is None
+
+    hints = product._extract_load_hints()
+    assert hints['crs'] == geometry.CRS('epsg:3857')
+    assert hints['resolution'] == (-10, 10)
+    assert 'align' not in hints
+
+    product = mk_sample_product('test_product',
+                                load=dict(crs='epsg:3857',
+                                          align={'x': 5, 'y': 6},
+                                          resolution={'x': 10, 'y': -10}))
+
+    hints = product.load_hints()
+    assert hints['output_crs'] == geometry.CRS('epsg:3857')
+    assert hints['resolution'] == (-10, 10)
+    assert hints['align'] == (6, 5)
+    assert product.default_crs == geometry.CRS('epsg:3857')
+    assert product.default_resolution == (-10, 10)
+    assert product.default_align == (6, 5)
+
+    product = mk_sample_product('test_product',
+                                load=dict(crs='epsg:4326',
+                                          align={'longitude': 0.5, 'latitude': 0.6},
+                                          resolution={'longitude': 1.2, 'latitude': -1.1}))
+
+    hints = product.load_hints()
+    assert hints['output_crs'] == geometry.CRS('epsg:4326')
+    assert hints['resolution'] == (-1.1, 1.2)
+    assert hints['align'] == (0.6, 0.5)
+
+    # check it's cached
+    assert product.load_hints() is product.load_hints()
+
+    # check schema: crs and resolution are compulsory
+    for k in ('resolution', 'crs'):
+        doc = deepcopy(product.definition)
+        assert DatasetType.validate(doc) is None
+
+        doc['load'].pop(k)
+        assert k not in doc['load']
+
+        with pytest.raises(InvalidDocException):
+            DatasetType.validate(doc)
+
+    # check GridSpec leakage doesn't happen for fully defined gridspec
+    product = mk_sample_product('test', with_grid_spec=True)
+    assert product.grid_spec is not None
+    assert product.load_hints() == {}
+
+    # check for fallback into partially defined `storage:`
+    product = mk_sample_product('test', storage=dict(
+        crs='EPSG:3857',
+        resolution={'x': 10, 'y': -10}))
+    assert product.grid_spec is None
+    assert product.default_resolution == (-10, 10)
+    assert product.default_crs == geometry.CRS('EPSG:3857')
+
+    # check for fallback into partially defined `storage:`
+    # no resolution -- no hints
+    product = mk_sample_product('test', storage=dict(
+        crs='EPSG:3857'))
+    assert product.grid_spec is None
+    assert product.load_hints() == {}
+
+    # check misspelled load hints
+    product = mk_sample_product('test_product',
+                                load=dict(crs='epsg:4326',
+                                          resolution={'longtude': 1.2, 'latitude': -1.1}))
+    assert product.load_hints() == {}
+
+
 def test_measurement():
     # Can create a measurement
     m = Measurement(name='t', dtype='uint8', nodata=255, units='1')
@@ -169,19 +250,21 @@ def test_measurement():
     assert 'dtype' in str(e.value)
 
 
-def test_like_geobox():
-    from datacube.testutils.geom import AlbersGS
-    from datacube.api.core import output_geobox
+def test_output_geobox_load_hints():
+    geobox0 = AlbersGS.tile_geobox((15, -40))
+
+    geobox = output_geobox(load_hints={'output_crs': geobox0.crs,
+                                       'resolution': geobox0.resolution},
+                           geopolygon=geobox0.extent)
+    assert geobox == geobox0
 
+
+def test_like_geobox():
     geobox = AlbersGS.tile_geobox((15, -40))
     assert output_geobox(like=geobox) is geobox
 
 
 def test_output_geobox_fail_paths():
-    from datacube.api.core import output_geobox
-    gs_nores = GridSpec(crs=geometry.CRS('EPSG:4326'),
-                        tile_size=None,
-                        resolution=None)
 
     with pytest.raises(ValueError):
         output_geobox()
@@ -189,8 +272,9 @@ def test_output_geobox_fail_paths():
     with pytest.raises(ValueError):
         output_geobox(output_crs='EPSG:4326')  # need resolution as well
 
+    # need bounds
     with pytest.raises(ValueError):
-        output_geobox(grid_spec=gs_nores)  # GridSpec with missing resolution
+        output_geobox(output_crs='EPSG:4326', resolution=(1, 1))
 
 
 def test_metadata_type():