Repository: rasterio/rasterio

Issue:
Open one dataset at a time during merge of GeoTIFFs
I suggest to change the behavior of rasterio.merge to always have only one dataset open. Opening all datasets at the same time can lead to Too many open files error mentioned in #1636. 

There are alternatives how this error could be fixed however none of them looks optimal to me. Firstly, raising operating system's limit on open file descriptors could solve this but it does not seem to me that this is good solution as I think we should not request this from our users. Secondly, merging GeoTIFFs in batches could also solve this error but that leads to longer run time (even double for big sets of GeoTIFFs).


Discussion:
@eseglem, as the author, what's your take? I think we both imagined merge would take a small number of datasets as input. Should it work with thousands?
On a related point, should this tool handle datasets that are too large to fit into memory? I'd blindly attempted to use it to merge a global set of tiles at 30m resolution, which works with `gdal_merge.py`, but fails in rasterio as an array of the required size can't be allocated in memory.

(This is possibly a somewhat wider question, and maybe I should make a different issue. I've not spotted any in-built support in rasterio for very large datasets - the underlying design currently seems to assume that any dataset can be handled all at once as a single numpy array)
In #1867 I propose to change the merge tool so that an open dataset can be passed in to serve as output. This should help with both a) very large output cases and b) improving the efficiency of merging in batches (reuse of the output across batches).
@DanielFEvans I'm having the same issue in that most of the rasters I am working with are far too large to load into memory all at once. Being able to read/write windows is very helpful, but higher-level functions like `merge` and `rasterize` don't really take advantage of them to the full extent. I feel like I'm writing the same code as other people in order to make this work, so I'm potentially keen to help with adding support for very large files if there's an appetite for it.

Resolving patch:
diff --git a/rasterio/merge.py b/rasterio/merge.py
--- a/rasterio/merge.py
+++ b/rasterio/merge.py
@@ -1,13 +1,15 @@
 """Copy valid pixels from input files to an output file."""
 
-
+from contextlib import contextmanager
 import logging
 import math
 import warnings
 
 import numpy as np
 
+import rasterio
 from rasterio import windows
+from rasterio.compat import string_types
 from rasterio.transform import Affine
 
 
@@ -33,7 +35,7 @@ def merge(datasets, bounds=None, res=None, nodata=None, dtype=None, precision=10
 
     Parameters
     ----------
-    datasets: list of dataset objects opened in 'r' mode
+    datasets : list of dataset objects opened in 'r' mode or filenames
         source datasets to be merged.
     bounds: tuple, optional
         Bounds of the output image (left, bottom, right, top).
@@ -94,23 +96,37 @@ def function(old_data, new_data, old_nodata, new_nodata, index=None, roff=None,
             out_transform: affine.Affine()
                 Information for mapping pixel coordinates in `dest` to another
                 coordinate system
-    """
-    first = datasets[0]
-    first_res = first.res
-    nodataval = first.nodatavals[0]
-    dt = first.dtypes[0]
 
+    """
     if method not in MERGE_METHODS and not callable(method):
         raise ValueError('Unknown method {0}, must be one of {1} or callable'
                          .format(method, MERGE_METHODS))
 
-    # Determine output band count
-    if indexes is None:
-        src_count = first.count
-    elif isinstance(indexes, int):
-        src_count = indexes
+    # Create a dataset_opener object to use in several places in this function.
+    if isinstance(datasets[0], string_types):
+        dataset_opener = rasterio.open
     else:
-        src_count = len(indexes)
+
+        @contextmanager
+        def nullcontext(obj):
+            try:
+                yield obj
+            finally:
+                pass
+
+        dataset_opener = nullcontext
+
+    with dataset_opener(datasets[0]) as first:
+        first_res = first.res
+        nodataval = first.nodatavals[0]
+        dt = first.dtypes[0]
+
+        if indexes is None:
+            src_count = first.count
+        elif isinstance(indexes, int):
+            src_count = indexes
+        else:
+            src_count = len(indexes)
 
     if not output_count:
         output_count = src_count
@@ -122,8 +138,9 @@ def function(old_data, new_data, old_nodata, new_nodata, index=None, roff=None,
         # scan input files
         xs = []
         ys = []
-        for src in datasets:
-            left, bottom, right, top = src.bounds
+        for dataset in datasets:
+            with dataset_opener(dataset) as src:
+                left, bottom, right, top = src.bounds
             xs.extend([left, right])
             ys.extend([bottom, top])
         dst_w, dst_s, dst_e, dst_n = min(xs), min(ys), max(xs), max(ys)
@@ -218,36 +235,43 @@ def copyto(old_data, new_data, old_nodata, new_nodata, **kwargs):
     else:
         raise ValueError(method)
 
-    for idx, src in enumerate(datasets):
-        # Real World (tm) use of boundless reads.
-        # This approach uses the maximum amount of memory to solve the
-        # problem. Making it more efficient is a TODO.
-
-        # 1. Compute spatial intersection of destination and source
-        src_w, src_s, src_e, src_n = src.bounds
-
-        int_w = src_w if src_w > dst_w else dst_w
-        int_s = src_s if src_s > dst_s else dst_s
-        int_e = src_e if src_e < dst_e else dst_e
-        int_n = src_n if src_n < dst_n else dst_n
-
-        # 2. Compute the source window
-        src_window = windows.from_bounds(
-            int_w, int_s, int_e, int_n, src.transform, precision=precision)
-        logger.debug("Src %s window: %r", src.name, src_window)
-
-        src_window = src_window.round_shape()
-
-        # 3. Compute the destination window
-        dst_window = windows.from_bounds(
-            int_w, int_s, int_e, int_n, output_transform, precision=precision)
-
-        # 4. Read data in source window into temp
-        trows, tcols = (
-            int(round(dst_window.height)), int(round(dst_window.width)))
-        temp_shape = (src_count, trows, tcols)
-        temp = src.read(out_shape=temp_shape, window=src_window,
-                        boundless=False, masked=True, indexes=indexes)
+    for idx, dataset in enumerate(datasets):
+        with dataset_opener(dataset) as src:
+            # Real World (tm) use of boundless reads.
+            # This approach uses the maximum amount of memory to solve the
+            # problem. Making it more efficient is a TODO.
+
+            # 1. Compute spatial intersection of destination and source
+            src_w, src_s, src_e, src_n = src.bounds
+
+            int_w = src_w if src_w > dst_w else dst_w
+            int_s = src_s if src_s > dst_s else dst_s
+            int_e = src_e if src_e < dst_e else dst_e
+            int_n = src_n if src_n < dst_n else dst_n
+
+            # 2. Compute the source window
+            src_window = windows.from_bounds(
+                int_w, int_s, int_e, int_n, src.transform, precision=precision
+            )
+            logger.debug("Src %s window: %r", src.name, src_window)
+
+            src_window = src_window.round_shape()
+
+            # 3. Compute the destination window
+            dst_window = windows.from_bounds(
+                int_w, int_s, int_e, int_n, output_transform, precision=precision
+            )
+
+            # 4. Read data in source window into temp
+            trows, tcols = (int(round(dst_window.height)), int(round(dst_window.width)))
+            temp_shape = (src_count, trows, tcols)
+            temp = src.read(
+                out_shape=temp_shape,
+                window=src_window,
+                boundless=False,
+                masked=True,
+                indexes=indexes,
+            )
 
         # 5. Copy elements of temp into dest
         roff, coff = (
diff --git a/rasterio/rio/merge.py b/rasterio/rio/merge.py
--- a/rasterio/rio/merge.py
+++ b/rasterio/rio/merge.py
@@ -49,30 +49,35 @@ def merge(ctx, files, output, driver, bounds, res, nodata, bidx, overwrite,
     output, files = resolve_inout(
         files=files, output=output, overwrite=overwrite)
 
-    with ctx.obj['env']:
-        datasets = [rasterio.open(f) for f in files]
-        dest, output_transform = merge_tool(datasets, bounds=bounds, res=res,
-                                            nodata=nodata, precision=precision,
-                                            indexes=(bidx or None))
-
-        profile = datasets[0].profile
-        profile['transform'] = output_transform
-        profile['height'] = dest.shape[1]
-        profile['width'] = dest.shape[2]
-        profile['driver'] = driver
-        profile['count'] = dest.shape[0]
-
-        if nodata is not None:
-            profile['nodata'] = nodata
-
-        profile.update(**creation_options)
-
-        with rasterio.open(output, 'w', **profile) as dst:
-            dst.write(dest)
-
-            # uses the colormap in the first input raster.
-            try:
-                colormap = datasets[0].colormap(1)
-                dst.write_colormap(1, colormap)
-            except ValueError:
-                pass
+    with ctx.obj["env"]:
+        dest, output_transform = merge_tool(
+            files,
+            bounds=bounds,
+            res=res,
+            nodata=nodata,
+            precision=precision,
+            indexes=(bidx or None),
+        )
+
+        with rasterio.open(files[0]) as first:
+            profile = first.profile
+            profile["transform"] = output_transform
+            profile["height"] = dest.shape[1]
+            profile["width"] = dest.shape[2]
+            profile["driver"] = driver
+            profile["count"] = dest.shape[0]
+
+            if nodata is not None:
+                profile["nodata"] = nodata
+
+            profile.update(**creation_options)
+
+            with rasterio.open(output, "w", **profile) as dst:
+                dst.write(dest)
+
+                # uses the colormap in the first input raster.
+                try:
+                    colormap = first.colormap(1)
+                    dst.write_colormap(1, colormap)
+                except ValueError:
+                    pass


Test patch:
diff --git a/tests/test_rio_merge.py b/tests/test_rio_merge.py
--- a/tests/test_rio_merge.py
+++ b/tests/test_rio_merge.py
@@ -556,3 +556,9 @@ def test_merge_precision(tmpdir, precision):
     result = runner.invoke(main_group, ["merge", "-f", "AAIGrid"] + precision + inputs + [outputname])
     assert result.exit_code == 0
     assert open(outputname).read() == textwrap.dedent(expected)
+
+
+def test_merge_filenames(tiffs):
+    inputs = [str(x) for x in tiffs.listdir()]
+    inputs.sort()
+    merge(inputs, res=2)