Repository: geopandas/geopandas

Issue:
Lazy Loading Dataframes
There have been times when I wanted to use geopandas to get an idea of what was in a large file. But, it took a long time to load it in. So, I used `fiona` instead. Are there any plans (or desires) to do lazy loading of the geopandas data?

This is library I recently saw is related:
https://vaex.readthedocs.io/en/latest/index.html
https://github.com/vaexio/vaex


Discussion:
What would be the ideal workflow for that? 
I suppose you used fiona to then only look at the first few elements? Or to look at the metadata? Or can you clarify what you want out of the "lazy" aspect?

In case of the first, we could maybe add a keyword to control how many records to fetch.

I am not fully sure geopandas would be a good fit for vaex. They eg use a lot of memory mapping, which is something that doesn't work with geometries. 
There were recently some talks again (and an old prototype) about combining geopandas with dask (see https://github.com/geopandas/geopandas/issues/461, https://github.com/dask/dask/issues/5438, https://github.com/geopandas/geopandas/pull/1143)
> I suppose you used fiona to then only look at the first few elements? Or to look at the metadata? Or can you clarify what you want out of the "lazy" aspect?

Something where I could look at the CRS and do a `head()` or `tail()` of a large dataset without having to load in the whole dataset. There are also times when I am only interested in a small subset of the file. 

I haven't worked out the details of how it would work under the hood yet, just at the initial idea stage at the moment.


Also, using vaex isn't necessary what I was thinking, just something that has implented some of these things that might be a good reference.
So, looking into this a bit more, I am thinking adding more options to `filter()` on open would be a good start.

Currently `geopandas.read_file()` supports `bbox`:
https://github.com/geopandas/geopandas/blob/e0b720580880464e4ed93cc269ac4d23b159feff/geopandas/io/file.py#L90

However, `filter()` also supports slices and geometries:
https://github.com/Toblerity/Fiona/blob/d6ec15f1c2e7a860bc57d35a51e36aeb48377a1a/fiona/collection.py#L231-L237

```
        """Returns an iterator over records, but filtered by a test for
        spatial intersection with the provided ``bbox``, a (minx, miny,
        maxx, maxy) tuple or a geometry ``mask``.

        Positional arguments ``stop`` or ``start, stop[, step]`` allows
        iteration to skip over items or stop at a specific item.
        """
```

So, maybe we could add the kwargs to `geopandas.read_file()`:
1. data_slice (start, stop[, step])
2. mask

This way, the user can specify what chunks they are interested in.

Thoughts?
I tested it out on a smaller dataset and it works nicely:
![image](https://user-images.githubusercontent.com/8699967/66725666-e53b0200-edf9-11e9-8eae-4438acdac471.png)

![image](https://user-images.githubusercontent.com/8699967/66725707-57134b80-edfa-11e9-99b0-8b20449d2387.png)


Resolving patch:
diff --git a/geopandas/io/file.py b/geopandas/io/file.py
--- a/geopandas/io/file.py
+++ b/geopandas/io/file.py
@@ -4,17 +4,21 @@
 import pandas as pd
 
 import fiona
-
-from geopandas import GeoDataFrame, GeoSeries
+from shapely.geometry import mapping
+from shapely.geometry.base import BaseGeometry
 
 try:
     from fiona import Env as fiona_env
 except ImportError:
     from fiona import drivers as fiona_env
+
+from geopandas import GeoDataFrame, GeoSeries
+
+
 # Adapted from pandas.io.common
 from urllib.request import urlopen as _urlopen
 from urllib.parse import urlparse as parse_url
-from urllib.parse import uses_relative, uses_netloc, uses_params
+from urllib.parse import uses_netloc, uses_params, uses_relative
 
 
 _FIONA18 = LooseVersion(fiona.__version__) >= LooseVersion("1.8")
@@ -30,18 +34,29 @@ def _is_url(url):
         return False
 
 
-def read_file(filename, bbox=None, **kwargs):
+def read_file(filename, bbox=None, mask=None, rows=None, **kwargs):
     """
     Returns a GeoDataFrame from a file or URL.
 
+    .. versionadded:: 0.7.0 mask, rows
+
     Parameters
     ----------
     filename: str
         Either the absolute or relative path to the file or URL to
         be opened.
-    bbox : tuple | GeoDataFrame or GeoSeries, default None
-        Filter features by given bounding box, GeoSeries, or GeoDataFrame.
+    bbox: tuple | GeoDataFrame or GeoSeries | shapely Geometry, default None
+        Filter features by given bounding box, GeoSeries, GeoDataFrame or a
+        shapely geometry. CRS mis-matches are resolved if given a GeoSeries
+        or GeoDataFrame. Cannot be used with mask.
+    mask: dict | GeoDataFrame or GeoSeries | shapely Geometry, default None
+        Filter for features that intersect with the given dict-like geojson
+        geometry, GeoSeries, GeoDataFrame or shapely geometry.
         CRS mis-matches are resolved if given a GeoSeries or GeoDataFrame.
+        Cannot be used with bbox.
+    rows: int or slice, default None
+        Load in specific rows by passing an integer (first `n` rows) or a
+        slice() object.
     **kwargs:
         Keyword args to be passed to the `open` or `BytesCollection` method
         in the fiona library when opening the file. For more information on
@@ -54,7 +69,7 @@ def read_file(filename, bbox=None, **kwargs):
 
     Returns
     -------
-    geodataframe : GeoDataFrame
+    :obj:`geopandas.GeoDataFrame`
 
     Notes
     -----
@@ -82,11 +97,29 @@ def read_file(filename, bbox=None, **kwargs):
                 else features.crs_wkt
             )
 
+            # handle loading the bounding box
             if bbox is not None:
-                if isinstance(bbox, GeoDataFrame) or isinstance(bbox, GeoSeries):
+                if isinstance(bbox, (GeoDataFrame, GeoSeries)):
                     bbox = tuple(bbox.to_crs(crs).total_bounds)
+                elif isinstance(bbox, BaseGeometry):
+                    bbox = bbox.bounds
                 assert len(bbox) == 4
-                f_filt = features.filter(bbox=bbox)
+            # handle loading the mask
+            elif isinstance(mask, (GeoDataFrame, GeoSeries)):
+                mask = mapping(mask.to_crs(crs).unary_union)
+            elif isinstance(mask, BaseGeometry):
+                mask = mapping(mask)
+            # setup the data loading filter
+            if rows is not None:
+                if isinstance(rows, int):
+                    rows = slice(rows)
+                elif not isinstance(rows, slice):
+                    raise TypeError("'rows' must be an integer or a slice.")
+                f_filt = features.filter(
+                    rows.start, rows.stop, rows.step, bbox=bbox, mask=mask
+                )
+            elif any((bbox, mask)):
+                f_filt = features.filter(bbox=bbox, mask=mask)
             else:
                 f_filt = features
 


Test patch:
diff --git a/geopandas/io/tests/test_file.py b/geopandas/io/tests/test_file.py
--- a/geopandas/io/tests/test_file.py
+++ b/geopandas/io/tests/test_file.py
@@ -5,9 +5,10 @@
 import pathlib
 import sys
 
-import fiona
 import numpy as np
 import pandas as pd
+
+import fiona
 from shapely.geometry import Point, Polygon, box
 
 import geopandas
@@ -17,6 +18,7 @@
 
 from geopandas.testing import assert_geodataframe_equal, assert_geoseries_equal
 from geopandas.tests.util import PACKAGE_DIR, validate_boro_df
+
 import pytest
 
 
@@ -272,6 +274,47 @@ def test_read_file_filtered(df_nybb):
     assert filtered_df_shape == (2, 5)
 
 
+def test_read_file_filtered__rows(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    filtered_df = read_file(nybb_filename, rows=1)
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (1, 5)
+
+
+def test_read_file_filtered__rows_bbox(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    bbox = (
+        1031051.7879884212,
+        224272.49231459625,
+        1047224.3104931959,
+        244317.30894023244,
+    )
+    filtered_df = read_file(nybb_filename, bbox=bbox, rows=slice(-1, None))
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (1, 5)
+
+
+def test_read_file_filtered__rows_bbox__polygon(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    bbox = box(
+        1031051.7879884212, 224272.49231459625, 1047224.3104931959, 244317.30894023244
+    )
+    filtered_df = read_file(nybb_filename, bbox=bbox, rows=slice(-1, None))
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (1, 5)
+
+
+def read_file_filtered_rows_invalid():
+    with pytest.raises(TypeError):
+        read_file(geopandas.datasets.get_path("nybb"), rows="not_a_slice")
+
+
 def test_read_file_filtered_with_gdf_boundary(df_nybb):
     full_df_shape = df_nybb.shape
     nybb_filename = geopandas.datasets.get_path("nybb")
@@ -292,6 +335,28 @@ def test_read_file_filtered_with_gdf_boundary(df_nybb):
     assert filtered_df_shape == (2, 5)
 
 
+def test_read_file_filtered_with_gdf_boundary__mask(df_nybb):
+    gdf_mask = geopandas.read_file(geopandas.datasets.get_path("naturalearth_lowres"))
+    gdf = geopandas.read_file(
+        geopandas.datasets.get_path("naturalearth_cities"),
+        mask=gdf_mask[gdf_mask.continent == "Africa"],
+    )
+    filtered_df_shape = gdf.shape
+    assert filtered_df_shape == (50, 2)
+
+
+def test_read_file_filtered_with_gdf_boundary__mask__polygon(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    mask = box(
+        1031051.7879884212, 224272.49231459625, 1047224.3104931959, 244317.30894023244
+    )
+    filtered_df = read_file(nybb_filename, mask=mask)
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (2, 5)
+
+
 def test_read_file_filtered_with_gdf_boundary_mismatched_crs(df_nybb):
     full_df_shape = df_nybb.shape
     nybb_filename = geopandas.datasets.get_path("nybb")
@@ -313,6 +378,27 @@ def test_read_file_filtered_with_gdf_boundary_mismatched_crs(df_nybb):
     assert filtered_df_shape == (2, 5)
 
 
+def test_read_file_filtered_with_gdf_boundary_mismatched_crs__mask(df_nybb):
+    full_df_shape = df_nybb.shape
+    nybb_filename = geopandas.datasets.get_path("nybb")
+    mask = geopandas.GeoDataFrame(
+        geometry=[
+            box(
+                1031051.7879884212,
+                224272.49231459625,
+                1047224.3104931959,
+                244317.30894023244,
+            )
+        ],
+        crs=CRS,
+    )
+    mask.to_crs(epsg=4326, inplace=True)
+    filtered_df = read_file(nybb_filename, mask=mask.geometry)
+    filtered_df_shape = filtered_df.shape
+    assert full_df_shape != filtered_df_shape
+    assert filtered_df_shape == (2, 5)
+
+
 def test_read_file_empty_shapefile(tmpdir):
 
     # create empty shapefile