Repository: microsoft/torchgeo

Issue:
GridGeoSampler may never reach end of image
In ran into this issue after running multiple tests and double-checking why my inference script using GridGeoSampler never reached the end (right and bottom) of an image. 

I'm using GridGeoSampler with an image of size 1591 x 2187 pixels, a chip size of 512 and stride of 256. For the sake of clarity, let's just say my image is 600 x 600 (pixel units or crs units doesn't matter here -> just assume minimum bounds in x and y are 0 if you're thinking crs units):

I want GridGeoSampler to sample my 600 x 600 image in chips of 512x512, with a stride of 256. 
The total number of row and columns [as calculated in the \_\_init__ and \_\_iter_ of GridGeoSampler](https://github.com/microsoft/torchgeo/blob/main/torchgeo/samplers/single.py#L198) will be:
rows = columns = (600 - 512) // 256 +1 = 1

Therefore, the \_\_iter__ function will use two for loops [with range(rows)](https://github.com/microsoft/torchgeo/blob/main/torchgeo/samplers/single.py#L205) and range (columns). Since those two variables are 1, only 1 512x512 sample will be returned by the sampler, thus ignoring 88 (600-512) pixels of width and height where valid values existed in my image.

To adapt the current implementation, I believe rows and columns should be calculated this way:
( image width - chip size + stride size ) // stride size + 1
or in the code, for rows:
`rows = int((bounds.maxy - bounds.miny - self.size[0] + self.stride[0]) // self.stride[0]) + 1`


Discussion:
As a result of #376, GridGeoSampler will also skip areas smaller than the patch size entirely.

I think the issue you reported is easy to fix, we just have to be careful not to go out of bounds of the image. So the minx/miny will need to be adjusted with a smaller stride for the last sample per row/col.
> As a result of #376, GridGeoSampler will also skip areas smaller than the patch size entirely.
> 
> I think the issue you reported is easy to fix, we just have to be careful not to go out of bounds of the image. So the minx/miny will need to be adjusted with a smaller stride for the last sample per row/col.

Hadn't though of the "smaller stride at the end of image" solution. I guess this ensures the chip_size is respected and could definitely work. The only drawback is that this changes the expected behaviour of the sampler since the stride may not always be the size that was initially set (in my dummy example, the last stride would be 88 rather than the expected 256). 

If a user (like me :) )  is stitching inference chips into a large array assuming the stride is constant (let's say he's using pixel units from #279 ), the result of the "smaller stride at the end of image" solution may be surprising. If this solution is chosen, I guess the user will have to make sure he/she is not naively assuming the stride will always be constant and make sure to rely on the bbox information returned by the sampler either directly from the \_\_iter__ function of the sampler of from a \_\_getitem__ as in [RasterDataset](https://github.com/microsoft/torchgeo/blob/6bc3779907ef4a92952763ef338a471425bcce36/torchgeo/datasets/geo.py#L432).

An alternative that I'd consider is to keep a constant stride, but return a sample that is truncated (in my example: a 88 x 512 sample for the last column and vice versa for last row), leaving it to the user to use a pad_to function [as is done in test_transforms on the Chesapeake dataset](https://github.com/microsoft/torchgeo/blob/744078fcff7e48957ffa5d0cd6b8acf3f5767b0a/torchgeo/datamodules/chesapeake.py#L256).

To synthesize:
Solution 1: "smaller stride at the end of image, but no change in chip size value"
Solution 2: "truncated chip at the end of image, but no change to stride size value"

In both cases, I'd issue a warning since either (1) the stride's value or (2) the chip size's value as inputted by the user will be forced to change at the limits of an image.

EDIT:
Just checked the discussion on issue #319. The conclusions of that discussion would logically lead to solution 1 rather than 2. I guess we'll have to assume the user is using the bbox returned by the sampler to perform stitching. I also see one potential flaw with 2: the grid sampling with GridGeoSampler may sample from right to left or bottom to top of an image if using CRS units. Therefore, the limit could potentially (and quite frequently) be at the top and left of image. If using a pad_to function as mentionned above, the padding function won't "know" the border the sample to pad is coming from and would naively assume (at least in the Cheseapeake implementation) last samples would be right and bottom borders. This, again, would cause problems at stitching since the padding value (generally 0) could overlap a region where valid values were predicted.

To help illustrate, I'll borrow from @adamjstewart's artwork :). Given two chips with a one pixel overlap, ie stride = chip_size - 1, where L: last, SL: second to last and 0: padding. Suppose these are being sampled from CRS units where maxx is on the left (pixel wise) of minx. 

```
+-----+
|    0|
|  L 0|
|    0|
+-----+
+-----+
|     |
|  SL |
|     |
+-----+
+----+-+----+
|    |x|    |
|  L |x| SL |  --­­> potential stitching problems
|    |x|    |
+----+-+----+
```
Also see #30. Part of the reason that GeoDatasets return the CRS and bbox is so you can stitch things back together easily even if the stride changes for the last image. Of course, we still need to actually write stitching utilities to use this information.
Thanks a ton for the bug report @remtav!

Here's a notebook that gives a self-contained example of:
- running a model on a RasterDataset made up of a single tile with GridGeoSampler (the model is a pre-trained land cover model I have laying around and the imagery is NAIP 2013 imagery from Maryland)
- stitching the results together in the inference loop
- visualizing the input/output

https://gist.github.com/calebrob6/7b226eb73877187f85fb5e1621bb7971 

![image](https://user-images.githubusercontent.com/1088078/155824198-aa587e37-2659-41e8-ba8e-e1628bbbce8e.png)

![image](https://user-images.githubusercontent.com/1088078/155824215-fb98caf7-82ce-45f2-9ed2-a96c4dd65b7e.png)

There is definitely something wrong here, but I haven't parsed it yet.

Resolving patch:
diff --git a/torchgeo/samplers/single.py b/torchgeo/samplers/single.py
--- a/torchgeo/samplers/single.py
+++ b/torchgeo/samplers/single.py
@@ -4,6 +4,7 @@
 """TorchGeo samplers."""
 
 import abc
+import math
 from typing import Callable, Iterable, Iterator, Optional, Tuple, Union
 
 import torch
@@ -146,7 +147,7 @@ def __len__(self) -> int:
 
 
 class GridGeoSampler(GeoSampler):
-    """Samples elements in a grid-like fashion.
+    r"""Samples elements in a grid-like fashion.
 
     This is particularly useful during evaluation when you want to make predictions for
     an entire region of interest. You want to minimize the amount of redundant
@@ -158,6 +159,21 @@ class GridGeoSampler(GeoSampler):
     The overlap between each chip (``chip_size - stride``) should be approximately equal
     to the `receptive field <https://distill.pub/2019/computing-receptive-fields/>`_ of
     the CNN.
+
+    Note that the stride of the final set of chips in each row/column may be adjusted so
+    that the entire :term:`tile` is sampled without exceeding the bounds of the dataset.
+
+    Let :math:`i` be the size of the input tile. Let :math:`k` be the requested size of
+    the output patch. Let :math:`s` be the requested stride. Let :math:`o` be the number
+    of output rows/columns sampled from each tile. :math:`o` can then be computed as:
+
+    .. math::
+
+       o = \left\lceil \frac{i - k}{s} \right\rceil + 1
+
+    This is almost identical to relationship 5 in
+    https://doi.org/10.48550/arXiv.1603.07285. However, we use ceiling instead of floor
+    because we want to include the final remaining chip.
     """
 
     def __init__(
@@ -200,8 +216,8 @@ def __init__(
         for hit in self.index.intersection(tuple(self.roi), objects=True):
             bounds = BoundingBox(*hit.bounds)
             if (
-                bounds.maxx - bounds.minx > self.size[1]
-                and bounds.maxy - bounds.miny > self.size[0]
+                bounds.maxx - bounds.minx >= self.size[1]
+                and bounds.maxy - bounds.miny >= self.size[0]
             ):
                 self.hits.append(hit)
 
@@ -209,8 +225,14 @@ def __init__(
         for hit in self.hits:
             bounds = BoundingBox(*hit.bounds)
 
-            rows = int((bounds.maxy - bounds.miny - self.size[0]) // self.stride[0]) + 1
-            cols = int((bounds.maxx - bounds.minx - self.size[1]) // self.stride[1]) + 1
+            rows = (
+                math.ceil((bounds.maxy - bounds.miny - self.size[0]) / self.stride[0])
+                + 1
+            )
+            cols = (
+                math.ceil((bounds.maxx - bounds.minx - self.size[1]) / self.stride[1])
+                + 1
+            )
             self.length += rows * cols
 
     def __iter__(self) -> Iterator[BoundingBox]:
@@ -223,8 +245,14 @@ def __iter__(self) -> Iterator[BoundingBox]:
         for hit in self.hits:
             bounds = BoundingBox(*hit.bounds)
 
-            rows = int((bounds.maxy - bounds.miny - self.size[0]) // self.stride[0]) + 1
-            cols = int((bounds.maxx - bounds.minx - self.size[1]) // self.stride[1]) + 1
+            rows = (
+                math.ceil((bounds.maxy - bounds.miny - self.size[0]) / self.stride[0])
+                + 1
+            )
+            cols = (
+                math.ceil((bounds.maxx - bounds.minx - self.size[1]) / self.stride[1])
+                + 1
+            )
 
             mint = bounds.mint
             maxt = bounds.maxt
@@ -233,11 +261,17 @@ def __iter__(self) -> Iterator[BoundingBox]:
             for i in range(rows):
                 miny = bounds.miny + i * self.stride[0]
                 maxy = miny + self.size[0]
+                if maxy > bounds.maxy:
+                    maxy = bounds.maxy
+                    miny = bounds.maxy - self.size[0]
 
                 # For each column...
                 for j in range(cols):
                     minx = bounds.minx + j * self.stride[1]
                     maxx = minx + self.size[1]
+                    if maxx > bounds.maxx:
+                        maxx = bounds.maxx
+                        minx = bounds.maxx - self.size[1]
 
                     yield BoundingBox(minx, maxx, miny, maxy, mint, maxt)
 


Test patch:
diff --git a/tests/samplers/test_single.py b/tests/samplers/test_single.py
--- a/tests/samplers/test_single.py
+++ b/tests/samplers/test_single.py
@@ -182,9 +182,9 @@ def test_iter(self, sampler: GridGeoSampler) -> None:
             )
 
     def test_len(self, sampler: GridGeoSampler) -> None:
-        rows = ((100 - sampler.size[0]) // sampler.stride[0]) + 1
-        cols = ((100 - sampler.size[1]) // sampler.stride[1]) + 1
-        length = rows * cols * 2
+        rows = math.ceil((100 - sampler.size[0]) / sampler.stride[0]) + 1
+        cols = math.ceil((100 - sampler.size[1]) / sampler.stride[1]) + 1
+        length = rows * cols * 2  # two items in dataset
         assert len(sampler) == length
 
     def test_roi(self, dataset: CustomGeoDataset) -> None:
@@ -194,12 +194,35 @@ def test_roi(self, dataset: CustomGeoDataset) -> None:
             assert query in roi
 
     def test_small_area(self) -> None:
+        ds = CustomGeoDataset()
+        ds.index.insert(0, (0, 1, 0, 1, 0, 1))
+        sampler = GridGeoSampler(ds, 2, 10)
+        assert len(sampler) == 0
+
+    def test_tiles_side_by_side(self) -> None:
         ds = CustomGeoDataset()
         ds.index.insert(0, (0, 10, 0, 10, 0, 10))
-        ds.index.insert(1, (20, 21, 20, 21, 20, 21))
+        ds.index.insert(0, (0, 10, 10, 20, 0, 10))
         sampler = GridGeoSampler(ds, 2, 10)
-        for _ in sampler:
-            continue
+        for bbox in sampler:
+            assert bbox.area > 0
+
+    def test_integer_multiple(self) -> None:
+        ds = CustomGeoDataset()
+        ds.index.insert(0, (0, 10, 0, 10, 0, 10))
+        sampler = GridGeoSampler(ds, 10, 10, units=Units.CRS)
+        iterator = iter(sampler)
+        assert len(sampler) == 1
+        assert next(iterator) == BoundingBox(0, 10, 0, 10, 0, 10)
+
+    def test_float_multiple(self) -> None:
+        ds = CustomGeoDataset()
+        ds.index.insert(0, (0, 6, 0, 5, 0, 10))
+        sampler = GridGeoSampler(ds, 5, 5, units=Units.CRS)
+        iterator = iter(sampler)
+        assert len(sampler) == 2
+        assert next(iterator) == BoundingBox(0, 5, 0, 5, 0, 10)
+        assert next(iterator) == BoundingBox(1, 6, 0, 5, 0, 10)
 
     @pytest.mark.slow
     @pytest.mark.parametrize("num_workers", [0, 1, 2])