Repository: geopandas/geopandas

Issue:
Lazy Loading Dataframes
There have been times when I wanted to use geopandas to get an idea of what was in a large file. But, it took a long time to load it in. So, I used `fiona` instead. Are there any plans (or desires) to do lazy loading of the geopandas data?

This is library I recently saw is related:
https://vaex.readthedocs.io/en/latest/index.html
https://github.com/vaexio/vaex

Discussion:
martinfleis: I like this!

`slice` looks perfect. In terms of `mask`, which expects geometry (correct?), what can we pass? GeoDataFrame, GeoSeries, Polygon?? Can we support all of them?
snowman2: Sounds good 👍

I wasn't sure that to do about the geoseries for the mask.  Maybe just do a union of the geometries?
martinfleis: > Maybe just do a union of the geometries?

Yep. `geometry.unary_union` will do.
snowman2: Sounds like a good plan to me. I will update it with that change later.
codecov[bot]: # [Codecov](https://codecov.io/gh/geopandas/geopandas/pull/1160?src=pr&el=h1) Report
> Merging [#1160](https://codecov.io/gh/geopandas/geopandas/pull/1160?src=pr&el=desc) into [master](https://codecov.io/gh/geopandas/geopandas/commit/32609cb4cc3b60ea9cff864188e10bcaebd44bd3?src=pr&el=desc) will **decrease** coverage by `0.12%`.
> The diff coverage is `57.14%`.

[![Impacted file tree graph](https://codecov.io/gh/geopandas/geopandas/pull/1160/graphs/tree.svg?width=650&token=6TOPMcCBhB&height=150&src=pr)](https://codecov.io/gh/geopandas/geopandas/pull/1160?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1160      +/-   ##
==========================================
- Coverage   90.19%   90.06%   -0.13%     
==========================================
  Files          21       21              
  Lines        2039     2044       +5     
==========================================
+ Hits         1839     1841       +2     
- Misses        200      203       +3
```


| [Impacted Files](https://codecov.io/gh/geopandas/geopandas/pull/1160?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [geopandas/io/file.py](https://codecov.io/gh/geopandas/geopandas/pull/1160/diff?src=pr&el=tree#diff-Z2VvcGFuZGFzL2lvL2ZpbGUucHk=) | `91.83% <57.14%> (-2.79%)` | :arrow_down: |
| [geopandas/tools/sjoin.py](https://codecov.io/gh/geopandas/geopandas/pull/1160/diff?src=pr&el=tree#diff-Z2VvcGFuZGFzL3Rvb2xzL3Nqb2luLnB5) | `97.36% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/geopandas/geopandas/pull/1160?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/geopandas/geopandas/pull/1160?src=pr&el=footer). Last update [32609cb...cbaac21](https://codecov.io/gh/geopandas/geopandas/pull/1160?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

snowman2: Hmmm:

```
RemoveError: 'setuptools' is a dependency of conda and cannot be removed from

conda's operating environment.

The command "conda update conda" failed and exited with 1 during .
```
martinfleis: @snowman2 known issue https://github.com/conda/conda/issues/9337
jorisvandenbossche: Some questions:

- How does the mask work? Is it intersects / contains / ... based? Is it using the actual shape or just the bounding box?
- Can you show a few benchmarks? Eg check the timing of reading the full file vs masking or slicing a small portion of it?

snowman2: > How does the mask work? 

The mask uses the setspatialfilter: https://gdal.org/doxygen/classOGRLayer.html#a0b4ab45cf97cbc470f0d60474d3e4169
```
This method set the geometry to be used as a spatial filter when fetching features via the GetNextFeature() method. Only features that geometrically intersect the filter geometry will be returned.

Currently this test is may be inaccurately implemented, but it is guaranteed that all features whose envelope (as returned by OGRGeometry::getEnvelope()) overlaps the envelope of the spatial filter will be returned. This can result in more shapes being returned that should strictly be the case.

Starting with GDAL 2.3, features with null or empty geometries will never be considered as matching a spatial filter.

This method makes an internal copy of the passed geometry. The passed geometry remains the responsibility of the caller, and may be safely destroyed.

For the time being the passed filter geometry should be in the same SRS as the layer (as returned by OGRLayer::GetSpatialRef()). In the future this may be generalized.
```


snowman2: Slicing benchmark:
https://github.com/geopandas/geopandas/issues/1157#issuecomment-541480982
![image](https://user-images.githubusercontent.com/8699967/66725666-e53b0200-edf9-11e9-8eae-4438acdac471.png)
martinfleis: @snowman2 can you resolve merge conflicts?
snowman2: @martinfleis, just resolved the conflicts.
snowman2: Looks like everything related passes. Unrelated failures in the build:

```
- TestSetitem.test_setitem_slice[True] 
- test_setitem_loc_iloc_slice
- test_setitem_slice
```
jorisvandenbossche: Sorry to come back once more on the keyword names, but: the thing I don't really like about `row_filter` is that it is not a "filter" in general (at least not in the sense that I explain pandas indexing, where you would typically "filter" with a boolean mask based on a condition). 

Other proposal: we could also call it `nrows` and allow `nrows=n` (which would be expanded to a slice of `slice(0, n)` to read the first n rows). 
This makes the usage of "reading the first n rows" (which I *suppose* will be the most common case?) easier, and consistent with how that is done in pandas `read_` functions. And compared to pandas, the `nrows` keyword would in addition also support an actual slice object to not start from row 0 but read a number of rows somewhere in the middle. 
(this makes it of course somewhat inconsistent with the word "nrows", which is also not ideal though ..)
snowman2: How about `isel`? Or `isel_rows`? It would be analogous to pandas `.isel()`
jorisvandenbossche: I think `isel` is from xarray-terminology and not pandas?
martinfleis: Isn't `isel` from xarray rather than pandas? What about simple `rows`?


snowman2: > I think isel is from xarray-terminology and not pandas?

Oh yeah. mixing that in my head.
snowman2: > What about simple rows?

I like that :+1:
jorisvandenbossche: `rows` is fine for me as well. I think `nrows` has a small plus in being consistent with pandas, `rows` on the other hand might capture the slice a bit better.
martinfleis: >  allow nrows=n (which would be expanded to a slice of slice(0, n) to read the first n rows)

I quite like this idea as well, I can imagine that is something I would personally do and it is easier to do `rows=10` than remembering to do `rows=slice(0, 10)`.
snowman2: > allow nrows=n (which would be expanded to a slice of slice(0, n) to read the first n rows)

Just added that in.

Patch:
diff --git a/doc/source/io.rst b/doc/source/io.rst
index 170262a9..a09e93c3 100644
--- a/doc/source/io.rst
+++ b/doc/source/io.rst
@@ -14,7 +14,7 @@ Reading Spatial Data
 
 which returns a GeoDataFrame object. (This is possible because *geopandas* makes use of the great `fiona <http://fiona.readthedocs.io/en/latest/manual.html>`_ library, which in turn makes use of a massive open-source program called `GDAL/OGR <http://www.gdal.org/>`_ designed to facilitate spatial data transformations).
 
-Any arguments passed to ``read_file()`` after the file name will be passed directly to ``fiona.open``, which does the actual data importation. In general, ``read_file`` is pretty smart and should do what you want without extra arguments, but for more help, type::
+Any arguments passed to :func:`geopandas.read_file` after the file name will be passed directly to ``fiona.open``, which does the actual data importation. In general, :func:`geopandas.read_file` is pretty smart and should do what you want without extra arguments, but for more help, type::
 
     import fiona; help(fiona.open)
 
@@ -42,13 +42,77 @@ If there are multiple datasets in a folder in the ZIP file, you also have to spe
     zipfile = "zip:///Users/name/Downloads/gadm36_AFG_shp.zip!data/gadm36_AFG_1.shp"
 
 
-*geopandas* can also get data from a PostGIS database using the ``read_postgis()`` command.
+*geopandas* can also get data from a PostGIS database using the :func:`geopandas.read_postgis` command.
+
+
+Reading subsets of the data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since geopandas is powered by Fiona, which is powered by GDAL, you can take advantage of
+pre-filtering when loading in larger datasets. This can be done geospatially with a geometry
+or bounding box. You can also filter rows loaded with a slice. Read more at :func:`geopandas.read_file`.
+
+Geometry Filter
+^^^^^^^^^^^^^^^
+
+.. versionadded:: 0.7.0
+
+The geometry filter only loads data that intersects with the geometry.
+
+.. code-block:: python
+
+    gdf_mask = geopandas.read_file(
+        geopandas.datasets.get_path("naturalearth_lowres")
+    )
+    gdf = geopandas.read_file(
+        geopandas.datasets.get_path("naturalearth_cities"),
+        mask=gdf_mask[gdf_mask.continent=="Africa"],
+    )
+
+Bounding Box Filter
+^^^^^^^^^^^^^^^^^^^
+
+.. versionadded:: 0.1.0
+
+The bounding box filter only loads data that intersects with the bounding box.
+
+.. code-block:: python
+
+    bbox = (
+        1031051.7879884212, 224272.49231459625, 1047224.3104931959, 244317.30894023244
+    )
+    gdf = geopandas.read_file(
+        geopandas.datasets.get_path("nybb"),
+        bbox=bbox,
+    )
+
+Row Filter
+^^^^^^^^^^
+
+.. versionadded:: 0.7.0
+
+Filter the rows loaded in from the file using an integer (for the first n rows)
+or a slice object.
+
+.. code-block:: python
+
+    gdf = geopandas.read_file(
+        geopandas.datasets.get_path("naturalearth_lowres"),
+        rows=10,
+    )
+    gdf = geopandas.read_file(
+        geopandas.datasets.get_path("naturalearth_lowres"),
+        rows=slice(10, 20),
+    )
+
 
 
 Writing Spatial Data
 ---------------------
 
-GeoDataFrames can be exported to many different standard formats using the ``GeoDataFrame.to_file()`` method. For a full list of supported formats, type ``import fiona; fiona.supported_drivers``.
+GeoDataFrames can be exported to many different standard formats using the
+:meth:`geopandas.GeoDataFrame.to_file` method.
+For a full list of supported formats, type ``import fiona; fiona.supported_drivers``.
 
 **Writing to Shapefile**::
 
diff --git a/geopandas/io/file.py b/geopandas/io/file.py
index 8ab4c481..0eaa937b 100644
--- a/geopandas/io/file.py
+++ b/geopandas/io/file.py
@@ -4,17 +4,21 @@ import numpy as np
 import pandas as pd
 
 import fiona
-
-from geopandas import GeoDataFrame, GeoSeries
+from shapely.geometry import mapping
+from shapely.geometry.base import BaseGeometry
 
 try:
     from fiona import Env as fiona_env
 except ImportError:
     from fiona import drivers as fiona_env
+
+from geopandas import GeoDataFrame, GeoSeries
+
+
 # Adapted from pandas.io.common
 from urllib.request import urlopen as _urlopen
 from urllib.parse import urlparse as parse_url
-from urllib.parse import uses_relative, uses_netloc, uses_params
+from urllib.parse import uses_netloc, uses_params, uses_relative
 
 
 _FIONA18 = LooseVersion(fiona.__version__) >= LooseVersion("1.8")
@@ -30,18 +34,29 @@ def _is_url(url):
         return False
 
 
-def read_file(filename, bbox=None, **kwargs):
+def read_file(filename, bbox=None, mask=None, rows=None, **kwargs):
     """
     Returns a GeoDataFrame from a file or URL.
 
+    .. versionadded:: 0.7.0 mask, rows
+
     Parameters
     ----------
     filename: str
         Either the absolute or relative path to the file or URL to
         be opened.
-    bbox : tuple | GeoDataFrame or GeoSeries, default None
-        Filter features by given bounding box, GeoSeries, or GeoDataFrame.
+    bbox: tuple | GeoDataFrame or GeoSeries | shapely Geometry, default None
+        Filter features by given bounding box, GeoSeries, GeoDataFrame or a
+        shapely geometry. CRS mis-matches are resolved if given a GeoSeries
+        or GeoDataFrame. Cannot be used with mask.
+    mask: dict | GeoDataFrame or GeoSeries | shapely Geometry, default None
+        Filter for features that intersect with the given dict-like geojson
+        geometry, GeoSeries, GeoDataFrame or shapely geometry.
         CRS mis-matches are resolved if given a GeoSeries or GeoDataFrame.
+        Cannot be used with bbox.
+    rows: int or slice, default None
+        Load in specific rows by passing an integer (first `n` rows) or a
+        slice() object.
     **kwargs:
         Keyword args to be passed to the `open` or `BytesCollection` method
         in the fiona library when opening the file. For more information on
@@ -54,7 +69,7 @@ def read_file(filename, bbox=None, **kwargs):
 
     Returns
     -------
-    geodataframe : GeoDataFrame
+    :obj:`geopandas.GeoDataFrame`
 
     Notes
     -----
@@ -82,11 +97,29 @@ def read_file(filename, bbox=None, **kwargs):
                 else features.crs_wkt
             )
 
+            # handle loading the bounding box
             if bbox is not None:
-                if isinstance(bbox, GeoDataFrame) or isinstance(bbox, GeoSeries):
+                if isinstance(bbox, (GeoDataFrame, GeoSeries)):
                     bbox = tuple(bbox.to_crs(crs).total_bounds)
+                elif isinstance(bbox, BaseGeometry):
+                    bbox = bbox.bounds
                 assert len(bbox) == 4
-                f_filt = features.filter(bbox=bbox)
+            # handle loading the mask
+            elif isinstance(mask, (GeoDataFrame, GeoSeries)):
+                mask = mapping(mask.to_crs(crs).unary_union)
+            elif isinstance(mask, BaseGeometry):
+                mask = mapping(mask)
+            # setup the data loading filter
+            if rows is not None:
+                if isinstance(rows, int):
+                    rows = slice(rows)
+                elif not isinstance(rows, slice):
+                    raise TypeError("'rows' must be an integer or a slice.")
+                f_filt = features.filter(
+                    rows.start, rows.stop, rows.step, bbox=bbox, mask=mask
+                )
+            elif any((bbox, mask)):
+                f_filt = features.filter(bbox=bbox, mask=mask)
             else:
                 f_filt = features
 


Test patch:
diff --git a/geopandas/io/tests/test_file.py b/geopandas/io/tests/test_file.py
index eb1b7c0b..b2301b43 100644
--- a/geopandas/io/tests/test_file.py
+++ b/geopandas/io/tests/test_file.py
@@ -5,9 +5,10 @@ import os
 import pathlib
 import sys
 
-import fiona
 import numpy as np
 import pandas as pd
+
+import fiona
 from shapely.geometry import Point, Polygon, box
 
 import geopandas
@@ -17,6 +18,7 @@ from geopandas._compat import PANDAS_GE_024
 
 from geopandas.testing import assert_geodataframe_equal, assert_geoseries_equal
 from geopandas.tests.util import PACKAGE_DIR, validate_boro_df
+
 import pytest
 
 
@@ -272,6 +274,47 @@ def test_read_file_filtered(df_nybb):
     assert filtered_df_shape == (2, 5)
 
 
+def test_read_file_filtered__rows(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    filtered_df = read_file(nybb_filename, rows=1)
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (1, 5)
+
+
+def test_read_file_filtered__rows_bbox(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    bbox = (
+        1031051.7879884212,
+        224272.49231459625,
+        1047224.3104931959,
+        244317.30894023244,
+    )
+    filtered_df = read_file(nybb_filename, bbox=bbox, rows=slice(-1, None))
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (1, 5)
+
+
+def test_read_file_filtered__rows_bbox__polygon(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    bbox = box(
+        1031051.7879884212, 224272.49231459625, 1047224.3104931959, 244317.30894023244
+    )
+    filtered_df = read_file(nybb_filename, bbox=bbox, rows=slice(-1, None))
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (1, 5)
+
+
+def read_file_filtered_rows_invalid():
+    with pytest.raises(TypeError):
+        read_file(geopandas.datasets.get_path("nybb"), rows="not_a_slice")
+
+
 def test_read_file_filtered_with_gdf_boundary(df_nybb):
     full_df_shape = df_nybb.shape
     nybb_filename = geopandas.datasets.get_path("nybb")
@@ -292,6 +335,28 @@ def test_read_file_filtered_with_gdf_boundary(df_nybb):
     assert filtered_df_shape == (2, 5)
 
 
+def test_read_file_filtered_with_gdf_boundary__mask(df_nybb):
+    gdf_mask = geopandas.read_file(geopandas.datasets.get_path("naturalearth_lowres"))
+    gdf = geopandas.read_file(
+        geopandas.datasets.get_path("naturalearth_cities"),
+        mask=gdf_mask[gdf_mask.continent == "Africa"],
+    )
+    filtered_df_shape = gdf.shape
+    assert filtered_df_shape == (50, 2)
+
+
+def test_read_file_filtered_with_gdf_boundary__mask__polygon(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    mask = box(
+        1031051.7879884212, 224272.49231459625, 1047224.3104931959, 244317.30894023244
+    )
+    filtered_df = read_file(nybb_filename, mask=mask)
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (2, 5)
+
+
 def test_read_file_filtered_with_gdf_boundary_mismatched_crs(df_nybb):
     full_df_shape = df_nybb.shape
     nybb_filename = geopandas.datasets.get_path("nybb")
@@ -313,6 +378,27 @@ def test_read_file_filtered_with_gdf_boundary_mismatched_crs(df_nybb):
     assert filtered_df_shape == (2, 5)
 
 
+def test_read_file_filtered_with_gdf_boundary_mismatched_crs__mask(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    mask = geopandas.GeoDataFrame(
+        geometry=[
+            box(
+                1031051.7879884212,
+                224272.49231459625,
+                1047224.3104931959,
+                244317.30894023244,
+            )
+        ],
+        crs=CRS,
+    )
+    mask.to_crs(epsg=4326, inplace=True)
+    filtered_df = read_file(nybb_filename, mask=mask.geometry)
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (2, 5)
+
+
 def test_read_file_empty_shapefile(tmpdir):
 
     # create empty shapefile