Repository: opendatacube/datacube-core

Issue:
BUG: Scale in rectilinear reproject produces different results (without dask)
Related:
- #1448
- #1450

### Expected behaviour

reproject behaves similar to GDAL defaults

### Actual behaviour

The behavior has slight differences with GDAL defaults

### Steps to reproduce the behaviour
- Original CRS = EPSG:32615
- Original Transform = Affine(5.0, 0.0, 451560.0, 0.0, -5.0, 4640690.0)

```python
import datacube
import rioxarray
import xarray
from rasterio.enum import Resampling

dc = datacube.Datacube()
old = xarray.open_dataset("old_demsm.nc", decode_coords="all")  # loaded with datacube==1.8.12 
new = dc.load( # datacube==1.8.13 
    product=...,
    measurements=["demsm"],
    geom=geom,
    output_crs="EPSG:6933",
    resolution=[-10, 10],
    resampling="cubic_spline",
).squeeze()
```

![image](https://github.com/opendatacube/datacube-core/assets/8699967/28737a10-f813-4b6d-a374-22801875ef36)

```python
orignal_rasters = dc.find_datasets(
    product=...,
    measurements=["demsm"],
    geom=geom,
)
original_demsm = rioxarray.open_rasterio(
    orignal_rasters[0].raster_path,
    masked=True,
).rio.reproject_match(
    match_data_array=new["demsm"],
    resampling=Resampling.cubic_spline,
).rio.clip(
    [geom],
    crs="EPSG:4326",
).squeeze()
```
![image](https://github.com/opendatacube/datacube-core/assets/8699967/c9430328-60d4-414e-a17e-2a2c17035bc0)

### Environment information

* Which ``datacube --version`` are you using? 1.8.13

cc @alfredoahds 


Discussion:
[link](https://github.com/OSGeo/gdal/blob/5daa9c4eed6fa0c541f1f121a1b86ca5e11028b9/alg/gdalwarpkernel.cpp#L1037-L1062) to the code where GDAL calculates scales.

```
dfXScale = static_cast<double>(nDstXSize) / (nSrcXSize - dfSrcXExtraSize);
dfYScale = static_cast<double>(nDstYSize) / (nSrcYSize - dfSrcYExtraSize);
```
`scale2` is inverted from the original scale here: https://github.com/opendatacube/datacube-core/blob/c8795e3144fcf6cdabed23eecc9e8cb04626bcef/datacube/utils/geometry/tools.py#L650

If the `XSCALE` and `YSCALE` here:
https://github.com/opendatacube/datacube-core/blob/c8795e3144fcf6cdabed23eecc9e8cb04626bcef/datacube/storage/_read.py#L193

are adjusted to undo the invert and also remove the rounding to an integer:
```python
XSCALE=1 / rr.scale2[0],
YSCALE=1 / rr.scale2[1]
```
Then the difference is much smaller:
![image](https://github.com/opendatacube/datacube-core/assets/8699967/65e3be0c-17c3-428a-9d0e-31f51d490e5c)


Question: Any issues disabling scaling for non-dask flows? https://github.com/corteva/datacube-core/commit/ddd56e39f934f8f4e83281432e1f6a1e9892f3bd

Edit: Seems GDAL is not always correct? https://github.com/OSGeo/gdal/issues/7750#issuecomment-1564114854
If I add `dask_chunks={"time": 1, "x": 30, "y": 30},`

Without `XSCALE` | `YSCALE` set (`v1.18.12`):
![image](https://github.com/opendatacube/datacube-core/assets/8699967/48a2a9bb-5192-407b-aa34-2d4fa1df873f)

With` XSCALE=1 / rr.scale2[0]` & `YSCALE=1 / rr.scale2[1]`:

![image](https://github.com/opendatacube/datacube-core/assets/8699967/1f3f2736-e8be-48b3-af1f-90ba2244f386)

With `XSCALE=1` & `YSCALE=1`:
![image](https://github.com/opendatacube/datacube-core/assets/8699967/36ce31c7-a68c-457b-b9fd-d813e7abd851)

Current `master` | `v1.18.13`
![image](https://github.com/opendatacube/datacube-core/assets/8699967/910412dd-bde2-4c8b-8fe5-4cc291157812)

I think this is scale "direction error". `scale2=(2,2)` happens when down-sampling, whereas GDAL expects `XSCALE=0.5, YSCALE=0.5` for that use case.

Problem is in GDAL, problem is more obvious when `max(W,H)/min(W,H)` of the output image is high and when there is significant rotation component. I would not limit this to Dask only case. But obviously direction of scale needs to be 

1. Documented in datacube
2. Passed in correctly to GDAL

Honestly I disagree with the whole approach taken by GDAL here, it's attempting to be helpful by essentially injecting nearest neighbour shrink/expand step instead of sampling directly from source pixels. Which in itself is questionable approach, especially that this not really documented as far as I can tell. This is then made much worse by using approximation for computation of required shrink/expand factors that breaks down badly for tall and thin or short and wide destination images.

@SpacemanPaul I guess we'll need a new release after fixing this one.


  
My personal preference here would be to disable this shrink/expand step inside GDAL by always passing in `XSCALE=1, YSCALE=1` regardless of the scale change estimate.
Interesting, but honestly not unsurprising - the existing scale stuff was a bit of hack based on the metadata we had to hand (which was gathered for a completely different purpose.)

I'll try to do a bit more testing before release this time so we've got a better handle on what works best.
> My personal preference here would be to disable this shrink/expand step inside GDAL by always passing in `XSCALE=1, YSCALE=1` regardless of the scale change estimate.

The more I think about it, the more I tend to agree with you.  I think I need to look at the GDAL code and try to get a better understanding of what the undocumented XSCALE and YSCALE parameters actually do.

quote from issue linked above:

> They define respectively the ratio of the output width / input width and output height / input height, which is evaluated for each warping chunk, and such ratio is used to modulate the radius of the resampling kernel (when the ratio is for example 0.5, the effective radius of the bilinear resampling will go from its nominal 1 value to 2) .

I believe this is equivalent to inserting average based integer shrink of the source imagery followed by bilinear sampling from that. I tried looking at the code, but there are a lot of specialized implementations based on things like float vs int, sampling algorithm, whether it's a shrink or expand, nodata or not, opencl or cpu...

I'd start here: https://github.com/OSGeo/gdal/blob/master/alg/gdalwarpkernel.cpp#L1265



Ahhh that makes sense.  Does indeed sound like `XSCALE=1, YSCALE=1` is the safest approach for us (for now).  (and definitely less wrong than what I did in #1450)

Resolving patch:
diff --git a/datacube/storage/_read.py b/datacube/storage/_read.py
--- a/datacube/storage/_read.py
+++ b/datacube/storage/_read.py
@@ -129,7 +129,6 @@ def read_time_slice(rdr,
 
     is_nn = is_resampling_nn(resampling)
     scale = pick_read_scale(rr.scale, rdr)
-    scale_x, scale_y = [pick_read_scale(s) for s in rr.scale2]
 
     paste_ok, _ = can_paste(rr, ttol=0.9 if is_nn else 0.01)
 
@@ -183,14 +182,32 @@ def norm_read_args(roi, shape, extra_dim_index):
 
         pix = rdr.read(*norm_read_args(rr.roi_src, src_gbox.shape, extra_dim_index))
 
+        # XSCALE and YSCALE are (currently) undocumented arguments that rasterio passed through to
+        # GDAL.  Not using them results in very inaccurate warping in images with highly
+        # non-square (i.e. long and thin) aspect ratios.
+        #
+        # See https://github.com/OSGeo/gdal/issues/7750 as well as
+        # https://github.com/opendatacube/datacube-core/pull/1450 and
+        # https://github.com/opendatacube/datacube-core/issues/1456
+        #
+        # In theory we might be able to get better results for queries with significantly
+        # different vertical and horizontal scales, but explicitly using XSCALE=1, YSCALE=1
+        # appears to be most appropriate for most requests, and is demonstrably better
+        # than not setting them at all.
+        gdal_scale_params = {
+            "XSCALE": 1,
+            "YSCALE": 1,
+        }
         if rr.transform.linear is not None:
             A = (~src_gbox.transform)*dst_gbox.transform
             warp_affine(pix, dst, A, resampling,
-                        src_nodata=rdr.nodata, dst_nodata=dst_nodata)
+                        src_nodata=rdr.nodata, dst_nodata=dst_nodata,
+                        **gdal_scale_params)
         else:
+
             rio_reproject(pix, dst, src_gbox, dst_gbox, resampling,
                           src_nodata=rdr.nodata, dst_nodata=dst_nodata,
-                          XSCALE=scale_x, YSCALE=scale_y)
+                          **gdal_scale_params)
 
     return rr.roi_dst
 


Test patch:
diff --git a/tests/storage/test_storage.py b/tests/storage/test_storage.py
--- a/tests/storage/test_storage.py
+++ b/tests/storage/test_storage.py
@@ -233,7 +233,8 @@ def assert_same_read_results(source, dst_shape, dst_dtype, dst_transform, dst_no
                                 dst_transform=dst_transform,
                                 dst_crs=str(dst_projection),
                                 dst_nodata=dst_nodata,
-                                resampling=resampling)
+                                resampling=resampling,
+                                XSCALE=1, YSCALE=1)
 
     result = np.full(dst_shape, dst_nodata, dtype=dst_dtype)
     H, W = dst_shape