Repository: microsoft/torchgeo

Issue:
Download support broken for some GeoDatasets if directory does not exist
### Description

@adriantre this is a bug we (mostly I) introduced in #1442/#1597. The issue is that if `os.path.isdir(file)` is False, we automatically add the file to `self.files`, and then `self.files` evaluates to True in `_verify`. I think the safest solution is to not use `self.files` in `_verify`, but you might be able to think of a better solution.

I first noticed this during https://github.com/microsoft/torchgeo/pull/1442/commits/8ec2e932810f55128ca8c0ceb78152389d3c5f16 but didn't realize the impact of it until https://github.com/microsoft/torchgeo/pull/1604/commits/fe546bf160a02acf6275b3f97c1ecd94fec81490. I still want to get this release out so I figured I would give us something to do in 0.5.1. I don't know how many other datasets are impacted.

Current workaround is to create the directory before downloading. Not hard, just annoying.

### Steps to reproduce

```console
> python3 
>>> from torchgeo.datasets import ChesapeakeDE
>>> ChesapeakeDE('fake/path', download=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Adam/torchgeo/torchgeo/datasets/chesapeake.py", line 138, in __init__
    super().__init__(paths, crs, res, transforms=transforms, cache=cache)
  File "/Users/Adam/torchgeo/torchgeo/datasets/geo.py", line 434, in __init__
    raise FileNotFoundError(msg)
FileNotFoundError: No ChesapeakeDE data was found in `paths='fake/path''`
```

### Version

0.5.0


Discussion:
Before 0.5.0, each (individual) method `_verify` returned successfully if the file / all files exist locally. This check could be moved into `self.files`. I propose that `self.files` returns the set of files that exists locally; if none is found returns the empty set. I believe this is the same behaviour as [this line before 0.5.0](https://github.com/microsoft/torchgeo/blob/15c1b255ff1aaa3b98fe2d91bf854c2bbe3d46d8/torchgeo/datasets/geo.py#L333).

```python
# GeoDataset
@property
def files(self) -> set[str]:
    ...  # unchanged code removed

    files: set[str] = set()
    for path in paths:
        if os.path.exists(path):
            if os.path.isfile(path):
                files.add(path)
            else:  # meaning os.path.isdir(path) == True, since exists is checked
                pathname = os.path.join(path, "**", self.filename_glob)
                files |= set(glob.iglob(pathname, recursive=True))
        else:
            files |= self.handle_nonlocal_path(path)

    return files

@staticmethod
def handle_nonlocal_path(path: str) -> set:
    return set()
```

With this setup the user must make sure the files are available (as before 0.5.0). They can download files, unpack zips etc. Or they can implement their own logic by overriding `handle_nonlocal_path` to, for instance, point to a file within a zip-archive, point to a file in azure blob etc.

Proposed docstring:
> A set of all available files in the dataset.
>
> For all `paths`, finds local files that match `filename_glob`.
Recursively iterates any directory in `paths`.
Paths that do not match `filename_glob` in the local file system is by
default ignored. This behaviour may be overridden in
method `handle_nonlocal_path`.

Running `ChesapeakeDE('fake/path', download=True)` works with the above changes. I have not yet tested anything else.

I have a [branch reflecting these changes](https://github.com/adriantre/torchgeo/blob/1f25176bf932e33024fb8c585cd918d7fddad961/torchgeo/datasets/geo.py#L281). I'll wait with the PR until we have discussed the approach.
> I propose that self.files returns the set of files that exists locally; if none is found returns the empty set.

I don't like this because it makes it impossible to use remote/zip files without the user overriding `handle_nonlocal_path`. I'm under the impression that these things "just work" at the moment. As long as GDAL knows how to read remote/zip files, we don't need to confirm that they exist.

I think you already looked into this and found that it wasn't possible to confirm whether a file was remote/zip vs. local? Like the path may not always contain vsi or something like that. But will the path always contain a colon? What if we check both `os.path.isfile` and `os.path.isdir` for paths without a colon and trust the user for paths with a colon?

I agree that `files` should not return anything for an obviously local files/dirs that do not exist. Honestly, I would go further and turn it into an error message.

P.S. Would love to have better support for glob chars. Something like `paths="*.tif"` should work, but I don't think it does at the moment. So maybe keep that in mind during refactoring.
I did not realise until now the the `if dir else add`-logic was motivated by supporting vsi out of the box 😅 that is indeed a good feature.

There are two ways of specifying vsi-path, one without any commas. But I have looked at this before, something similar may be feasible.

I'll be back.
FYI I was able to use v0.5 to do COG reading with RasterDataset -- https://gist.github.com/calebrob6/df765911d86df6c648e99060222b1e0b


Resolving patch:
diff --git a/torchgeo/datasets/geo.py b/torchgeo/datasets/geo.py
--- a/torchgeo/datasets/geo.py
+++ b/torchgeo/datasets/geo.py
@@ -9,6 +9,7 @@
 import os
 import re
 import sys
+import warnings
 from collections.abc import Iterable, Sequence
 from typing import Any, Callable, Optional, Union, cast
 
@@ -29,7 +30,13 @@
 from torchvision.datasets import ImageFolder
 from torchvision.datasets.folder import default_loader as pil_loader
 
-from .utils import BoundingBox, concat_samples, disambiguate_timestamp, merge_samples
+from .utils import (
+    BoundingBox,
+    concat_samples,
+    disambiguate_timestamp,
+    merge_samples,
+    path_is_vsi,
+)
 
 
 class GeoDataset(Dataset[dict[str, Any]], abc.ABC):
@@ -298,8 +305,14 @@ def files(self) -> set[str]:
             if os.path.isdir(path):
                 pathname = os.path.join(path, "**", self.filename_glob)
                 files |= set(glob.iglob(pathname, recursive=True))
-            else:
+            elif os.path.isfile(path) or path_is_vsi(path):
                 files.add(path)
+            else:
+                warnings.warn(
+                    f"Could not find any relevant files for provided path '{path}'. "
+                    f"Path was ignored.",
+                    UserWarning,
+                )
 
         return files
 
diff --git a/torchgeo/datasets/utils.py b/torchgeo/datasets/utils.py
--- a/torchgeo/datasets/utils.py
+++ b/torchgeo/datasets/utils.py
@@ -737,3 +737,27 @@ def percentile_normalization(
         (img - lower_percentile) / (upper_percentile - lower_percentile + 1e-5), 0, 1
     )
     return img_normalized
+
+
+def path_is_vsi(path: str) -> bool:
+    """Checks if the given path is pointing to a Virtual File System.
+
+    .. note::
+       Does not check if the path exists, or if it is a dir or file.
+
+    VSI can for instance be Cloud Storage Blobs or zip-archives.
+    They will start with a prefix indicating this.
+    For examples of these, see references for the two accepted syntaxes.
+
+    * https://gdal.org/user/virtual_file_systems.html
+    * https://rasterio.readthedocs.io/en/latest/topics/datasets.html
+
+    Args:
+        path: string representing a directory or file
+
+    Returns:
+        True if path is on a virtual file system, else False
+
+    .. versionadded:: 0.6
+    """
+    return "://" in path or path.startswith("/vsi")


Test patch:
diff --git a/tests/datasets/test_geo.py b/tests/datasets/test_geo.py
--- a/tests/datasets/test_geo.py
+++ b/tests/datasets/test_geo.py
@@ -4,7 +4,7 @@
 import pickle
 from collections.abc import Iterable
 from pathlib import Path
-from typing import Union
+from typing import Optional, Union
 
 import pytest
 import torch
@@ -33,11 +33,13 @@ def __init__(
         bounds: BoundingBox = BoundingBox(0, 1, 2, 3, 4, 5),
         crs: CRS = CRS.from_epsg(4087),
         res: float = 1,
+        paths: Optional[Union[str, Iterable[str]]] = None,
     ) -> None:
         super().__init__()
         self.index.insert(0, tuple(bounds))
         self._crs = crs
         self.res = res
+        self.paths = paths or []
 
     def __getitem__(self, query: BoundingBox) -> dict[str, BoundingBox]:
         hits = self.index.intersection(tuple(query), objects=True)
@@ -152,6 +154,23 @@ def test_and_nongeo(self, dataset: GeoDataset) -> None:
         ):
             dataset & ds2  # type: ignore[operator]
 
+    def test_files_property_for_non_existing_file_or_dir(self, tmp_path: Path) -> None:
+        paths = [str(tmp_path), str(tmp_path / "non_existing_file.tif")]
+        with pytest.warns(UserWarning, match="Path was ignored."):
+            assert len(CustomGeoDataset(paths=paths).files) == 0
+
+    def test_files_property_for_virtual_files(self) -> None:
+        # Tests only a subset of schemes and combinations.
+        paths = [
+            "file://directory/file.tif",
+            "zip://archive.zip!folder/file.tif",
+            "az://azure_bucket/prefix/file.tif",
+            "/vsiaz/azure_bucket/prefix/file.tif",
+            "zip+az://azure_bucket/prefix/archive.zip!folder_in_archive/file.tif",
+            "/vsizip//vsiaz/azure_bucket/prefix/archive.zip/folder_in_archive/file.tif",
+        ]
+        assert len(CustomGeoDataset(paths=paths).files) == len(paths)
+
 
 class TestRasterDataset:
     @pytest.fixture(params=zip([["R", "G", "B"], None], [True, False]))