Repository: geopandas/geopandas
Pull request #2914 (2023-06-01 19:52:45)

REGR: fix read_file from urls
Fixes https://github.com/geopandas/geopandas/issues/2908

Alternative for https://github.com/geopandas/geopandas/pull/2912, since that didn't fix all cases of urls that no longer work. 
The approach in this PR is to only _not_ download the data ourselves ~if the url has an extension~ if the url indicates to support reading ranges (and in that case we assume the /vsicurl/ filesystem will work), and in all other cases still download the data as we did before, passing the raw bytes to pyogrio/fiona. 
If later pyogrio/fiona have better support for urls, we could again consider passing through all urls.


jorisvandenbossche: Given that there are also urls with extension that don't work out of the box with `/vsicurl`, I switched the approach here to check whether the URL supports partial reads (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Ranges). 
The idea is that `/vsicurl` is mostly (only?) useful in those cases, as it is especially important to be able read parts of large files. I verified that the original driving use case from https://github.com/geopandas/geopandas/issues/2795 (reading from large gpkg with spatial filter) works with this approach.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5f6fc4a0..16258cd4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,14 @@ New features and improvements:
 
 Bug fixes:
 
+## Version 0.13.1 (Jun 5, 2023)
+
+Bug fix:
+
+- Fix the a regression in reading from URLs using ``geopandas.read_file`` (#2908). This
+  restores the behaviour to download all data up-front before passing it to the
+  underlying engine (fiona or pyogrio), except if the server supports partial requests
+  (to support reading a subset of a large file).
 
 ## Version 0.13 (May 6, 2023)
 
diff --git a/geopandas/io/file.py b/geopandas/io/file.py
index 00c5f364..579bce42 100644
--- a/geopandas/io/file.py
+++ b/geopandas/io/file.py
@@ -14,7 +14,9 @@ from shapely.geometry.base import BaseGeometry
 from geopandas import GeoDataFrame, GeoSeries
 
 # Adapted from pandas.io.common
+from urllib.parse import urlparse as parse_url
 from urllib.parse import uses_netloc, uses_params, uses_relative
+import urllib.request
 
 
 _VALID_URLS = set(uses_relative + uses_netloc + uses_params)
@@ -149,6 +151,14 @@ def _expand_user(path):
     return path
 
 
+def _is_url(url):
+    """Check to see if *url* has a valid protocol."""
+    try:
+        return parse_url(url).scheme in _VALID_URLS
+    except Exception:
+        return False
+
+
 def _is_zip(path):
     """Check if a given path is a zipfile"""
     parsed = fiona.path.ParsedPath.from_uri(path)
@@ -228,16 +238,37 @@ def _read_file(filename, bbox=None, mask=None, rows=None, engine=None, **kwargs)
     The format drivers will attempt to detect the encoding of your data, but
     may fail. In this case, the proper encoding can be specified explicitly
     by using the encoding keyword parameter, e.g. ``encoding='utf-8'``.
+
+    When specifying a URL, geopandas will check if the server supports reading
+    partial data and in that case pass the URL as is to the underlying engine,
+    which will then use the network file system handler of GDAL to read from
+    the URL. Otherwise geopandas will download the data from the URL and pass
+    all data in-memory to the underlying engine.
+    If you need more control over how the URL is read, you can specify the
+    GDAL virtual filesystem manually (e.g. ``/vsicurl/https://...``). See the
+    GDAL documentation on filesystems for more details
+    (https://gdal.org/user/virtual_file_systems.html#vsicurl-http-https-ftp-files-random-access).
+
     """
     engine = _check_engine(engine, "'read_file' function")
 
     filename = _expand_user(filename)
 
+    from_bytes = False
+    if _is_url(filename):
+        # if it is a url that supports random access -> pass through to
+        # pyogrio/fiona as is (to support downloading only part of the file)
+        # otherwise still download manually because pyogrio/fiona don't support
+        # all types of urls (https://github.com/geopandas/geopandas/issues/2908)
+        request = urllib.request.urlopen(filename)
+        if not request.getheader("Accept-Ranges") == "bytes":
+            filename = request.read()
+            from_bytes = True
+
     if engine == "pyogrio":
         return _read_file_pyogrio(filename, bbox=bbox, mask=mask, rows=rows, **kwargs)
 
     elif engine == "fiona":
-        from_bytes = False
         if pd.api.types.is_file_like(filename):
             data = filename.read()
             path_or_bytes = data.encode("utf-8") if isinstance(data, str) else data


Test patch:
diff --git a/geopandas/io/tests/test_file.py b/geopandas/io/tests/test_file.py
index f4f08855..82457d59 100644
--- a/geopandas/io/tests/test_file.py
+++ b/geopandas/io/tests/test_file.py
@@ -557,21 +557,22 @@ def test_read_file(engine):
 
 
 @pytest.mark.web
-def test_read_file_remote_geojson_url(engine):
-    url = (
+@pytest.mark.parametrize(
+    "url",
+    [
+        # geojson url
         "https://raw.githubusercontent.com/geopandas/geopandas/"
-        "main/geopandas/tests/data/null_geom.geojson"
-    )
-    gdf = read_file(url, engine=engine)
-    assert isinstance(gdf, geopandas.GeoDataFrame)
-
-
-@pytest.mark.web
-def test_read_file_remote_zipfile_url(engine):
-    url = (
+        "main/geopandas/tests/data/null_geom.geojson",
+        # url to zip file
         "https://raw.githubusercontent.com/geopandas/geopandas/"
-        "main/geopandas/datasets/nybb_16a.zip"
-    )
+        "main/geopandas/datasets/nybb_16a.zip",
+        # url to zipfile without extension
+        "https://geonode.goosocean.org/download/480",
+        # url to web service
+        "https://demo.pygeoapi.io/stable/collections/obs/items",
+    ],
+)
+def test_read_file_url(engine, url):
     gdf = read_file(url, engine=engine)
     assert isinstance(gdf, geopandas.GeoDataFrame)