Repository: rasterio/rasterio

Issue:
fillnodata does not use nodata values
## Expected behavior and actual behavior.

I expected `fillnodata` to automatically use the `source.nodata` value, but it didn't. A workaround is to read the corresponding mask. (See https://github.com/mapbox/rasterio/pull/253#issuecomment-325963662 and on)

## Steps to reproduce the problem.

```
$ rio info N00E006.hgt 
{"bounds": [5.999583333333334, -0.0004166666666667318, 7.000416666666667, 1.0004166666666667], "colorinterp": ["undefined"], "count": 1, "crs": "EPSG:4326", "descriptions": [null], "driver": "SRTMHGT", "dtype": "int16", "height": 1201, "indexes": [1], "lnglat": [6.5, 0.5], "mask_flags": [["nodata"]], "nodata": -32768.0, "res": [0.0008333333333333334, 0.0008333333333333334], "shape": [1201, 1201], "tiled": false, "transform": [0.0008333333333333334, 0.0, 5.999583333333334, 0.0, -0.0008333333333333334, 1.0004166666666667, 0.0, 0.0, 1.0], "units": ["m"], "width": 1201}
$ gdalinfo N00E006.hgt 
Driver: SRTMHGT/SRTMHGT File Format
Files: N00E006.hgt
Size is 1201, 1201
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        TOWGS84[0,0,0,0,0,0,0],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9108"]],
    AUTHORITY["EPSG","4326"]]
Origin = (5.999583333333334,1.000416666666667)
Pixel Size = (0.000833333333333,-0.000833333333333)
Metadata:
  AREA_OR_POINT=Point
Corner Coordinates:
Upper Left  (   5.9995833,   1.0004167) (  5d59'58.50"E,  1d 0' 1.50"N)
Lower Left  (   5.9995833,  -0.0004167) (  5d59'58.50"E,  0d 0' 1.50"S)
Upper Right (   7.0004167,   1.0004167) (  7d 0' 1.50"E,  1d 0' 1.50"N)
Lower Right (   7.0004167,  -0.0004167) (  7d 0' 1.50"E,  0d 0' 1.50"S)
Center      (   6.5000000,   0.5000000) (  6d30' 0.00"E,  0d30' 0.00"N)
Band 1 Block=1201x1 Type=Int16, ColorInterp=Undefined
  NoData Value=-32768
  Unit Type: m
$ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rasterio
>>> source = rasterio.open("N00E006.hgt", 'r')
>>> source.nodata
-32768.0
>>> from rasterio.fill import fillnodata
>>> data = source.read(1)
>>> source.nodata in data
True
>>> source.nodata in fillnodata(data)
True
>>> (data == fillnodata(data)).all()
True
>>> source.nodata in fillnodata(data, mask=source.read_masks(1))
False
```

## Operating system

Linux Mint, 64bit

## Rasterio version and provenance

1.0a9 manylinux1 wheel installed from PyPI using pip.

```
$ conda list rasterio
# packages in environment at /home/juanlu/.miniconda36/envs/env:
#
rasterio                  1.0a9                     <pip>
$ pip list | grep rasterio
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
rasterio (1.0a9)
$ python -q
>>> import rasterio
>>> rasterio.__version__
'1.0a9'
```

fillnodata does not use nodata values
## Expected behavior and actual behavior.

I expected `fillnodata` to automatically use the `source.nodata` value, but it didn't. A workaround is to read the corresponding mask. (See https://github.com/mapbox/rasterio/pull/253#issuecomment-325963662 and on)

## Steps to reproduce the problem.

```
$ rio info N00E006.hgt 
{"bounds": [5.999583333333334, -0.0004166666666667318, 7.000416666666667, 1.0004166666666667], "colorinterp": ["undefined"], "count": 1, "crs": "EPSG:4326", "descriptions": [null], "driver": "SRTMHGT", "dtype": "int16", "height": 1201, "indexes": [1], "lnglat": [6.5, 0.5], "mask_flags": [["nodata"]], "nodata": -32768.0, "res": [0.0008333333333333334, 0.0008333333333333334], "shape": [1201, 1201], "tiled": false, "transform": [0.0008333333333333334, 0.0, 5.999583333333334, 0.0, -0.0008333333333333334, 1.0004166666666667, 0.0, 0.0, 1.0], "units": ["m"], "width": 1201}
$ gdalinfo N00E006.hgt 
Driver: SRTMHGT/SRTMHGT File Format
Files: N00E006.hgt
Size is 1201, 1201
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        TOWGS84[0,0,0,0,0,0,0],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9108"]],
    AUTHORITY["EPSG","4326"]]
Origin = (5.999583333333334,1.000416666666667)
Pixel Size = (0.000833333333333,-0.000833333333333)
Metadata:
  AREA_OR_POINT=Point
Corner Coordinates:
Upper Left  (   5.9995833,   1.0004167) (  5d59'58.50"E,  1d 0' 1.50"N)
Lower Left  (   5.9995833,  -0.0004167) (  5d59'58.50"E,  0d 0' 1.50"S)
Upper Right (   7.0004167,   1.0004167) (  7d 0' 1.50"E,  1d 0' 1.50"N)
Lower Right (   7.0004167,  -0.0004167) (  7d 0' 1.50"E,  0d 0' 1.50"S)
Center      (   6.5000000,   0.5000000) (  6d30' 0.00"E,  0d30' 0.00"N)
Band 1 Block=1201x1 Type=Int16, ColorInterp=Undefined
  NoData Value=-32768
  Unit Type: m
$ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rasterio
>>> source = rasterio.open("N00E006.hgt", 'r')
>>> source.nodata
-32768.0
>>> from rasterio.fill import fillnodata
>>> data = source.read(1)
>>> source.nodata in data
True
>>> source.nodata in fillnodata(data)
True
>>> (data == fillnodata(data)).all()
True
>>> source.nodata in fillnodata(data, mask=source.read_masks(1))
False
```

## Operating system

Linux Mint, 64bit

## Rasterio version and provenance

1.0a9 manylinux1 wheel installed from PyPI using pip.

```
$ conda list rasterio
# packages in environment at /home/juanlu/.miniconda36/envs/env:
#
rasterio                  1.0a9                     <pip>
$ pip list | grep rasterio
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
rasterio (1.0a9)
$ python -q
>>> import rasterio
>>> rasterio.__version__
'1.0a9'
```



Discussion:
@Juanlu001 ah, yes, I agree the docstring is misleading. From your example

```python
>>> data = source.read(1)
>>> filled = fillnodata(data)
```

the best we can do is to use the mask of `data` if it is an instance of `MaskedArray`, because there's no dataset in the function scope.

It was my original intent that `fillnodata()` be able to work on dataset bands as well as arrays, but this has never been tested and does not in fact work reliably. I'm inclined to remove that incomplete functionality and focus on filling arrays only.
@Juanlu001 ah, yes, I agree the docstring is misleading. From your example

```python
>>> data = source.read(1)
>>> filled = fillnodata(data)
```

the best we can do is to use the mask of `data` if it is an instance of `MaskedArray`, because there's no dataset in the function scope.

It was my original intent that `fillnodata()` be able to work on dataset bands as well as arrays, but this has never been tested and does not in fact work reliably. I'm inclined to remove that incomplete functionality and focus on filling arrays only.

Resolving patch:
diff --git a/rasterio/fill.py b/rasterio/fill.py
--- a/rasterio/fill.py
+++ b/rasterio/fill.py
@@ -3,6 +3,9 @@
 import rasterio
 from rasterio._fill import _fillnodata
 from rasterio.env import ensure_env
+from rasterio import dtypes
+
+from numpy.ma import MaskedArray
 
 
 @ensure_env
@@ -11,7 +14,7 @@ def fillnodata(
         mask=None,
         max_search_distance=100.0,
         smoothing_iterations=0):
-    """Fill holes in a raster dataset by interpolation from the edges.
+    """Fill holes in raster data by interpolation
 
     This algorithm will interpolate values for all designated nodata
     pixels (marked by zeros in `mask`). For each pixel a four direction
@@ -30,13 +33,16 @@ def fillnodata(
     Parameters
     ----------
     image : numpy ndarray
-        The source containing nodata holes.
+        The source image with holes to be filled. If a MaskedArray, the
+        inverse of its mask will define the pixels to be filled --
+        unless the ``mask`` argument is not None (see below).`
     mask : numpy ndarray or None
         A mask band indicating which pixels to interpolate. Pixels to
-        interpolate into are indicated by the value 0. Values > 0
-        indicate areas to use during interpolation. Must be same shape
-        as image. If `None`, a mask will be diagnosed from the source
-        data.
+        interpolate into are indicated by the value 0. Values
+        > 0 indicate areas to use during interpolation. Must be same
+        shape as image. This array always takes precedence over the
+        image's mask (see above). If None, the inverse of the image's
+        mask will be used if available.
     max_search_distance : float, optional
         The maxmimum number of pixels to search in all directions to
         find values to interpolate from. The default is 100.
@@ -49,6 +55,16 @@ def fillnodata(
     out : numpy ndarray
         The filled raster array.
     """
+    if mask is None and isinstance(image, MaskedArray):
+        mask = ~image.mask
+    if not dtypes.is_ndarray(mask):
+        raise ValueError("An mask array is required")
+
+    if isinstance(image, MaskedArray):
+        image = image.data
+    if not dtypes.is_ndarray(image):
+        raise ValueError("An image array is required")
+
     max_search_distance = float(max_search_distance)
     smoothing_iterations = int(smoothing_iterations)
     return _fillnodata(


Test patch:
diff --git a/tests/test_fillnodata.py b/tests/test_fillnodata.py
--- a/tests/test_fillnodata.py
+++ b/tests/test_fillnodata.py
@@ -1,23 +1,32 @@
-import logging
-import sys
+"""Tests of nodata filling"""
+
 import numpy as np
 import pytest
 
-import rasterio
 from rasterio.fill import fillnodata
 
-logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
 
-def test_fillnodata():
+@pytest.fixture(scope='session')
+def hole_in_ones():
+    """A 5x5 array with one nodata pixel dead center"""
+    a = np.ones((5, 5), dtype='uint8')
+    a[2][2] = 0
+    return a
+
+
+def test_fillnodata(hole_in_ones):
     """Test filling nodata values in an ndarray"""
-    # create a 5x5 array, with some missing data
-    a = np.ones([3, 3]) * 42
-    a[1][1] = 0
-    # find the missing data
-    mask = ~(a == 0)
-    # fill the missing data using interpolation from the edges
-    result = fillnodata(a, mask)
-    assert(np.all((np.ones([3, 3]) * 42) == result))
+    mask = hole_in_ones == 1
+    result = fillnodata(hole_in_ones, mask)
+    assert (result == 1).all()
+
+
+def test_fillnodata_masked_array(hole_in_ones):
+    """Test filling nodata values in a masked ndarray"""
+    ma = np.ma.masked_array(hole_in_ones, (hole_in_ones == 0))
+    result = fillnodata(ma)
+    assert (result == 1).all()
+
 
 def test_fillnodata_invalid_types():
     a = np.ones([3, 3])
@@ -26,20 +35,9 @@ def test_fillnodata_invalid_types():
     with pytest.raises(ValueError):
         fillnodata(a, 42)
 
-def test_fillnodata_mask_ones():
-    # when mask is all ones, image should be unmodified
-    a = np.ones([3, 3]) * 42
-    a[1][1] = 0
-    mask = np.ones([3, 3])
-    result = fillnodata(a, mask)
-    assert(np.all(a == result))
-
-'''
-def test_fillnodata_smooth():
-    a = np.array([[1,3,3,1],[2,0,0,2],[2,0,0,2],[1,3,3,1]], dtype=np.float64)
-    mask = ~(a == 0)
-    result = fillnodata(a, mask, max_search_distance=1, smoothing_iterations=0)
-    assert(result[1][1] == 3)
-    result = fillnodata(a, mask, max_search_distance=1, smoothing_iterations=1)
-    assert(round(result[1][1], 1) == 2.2)
-'''
+
+def test_fillnodata_mask_ones(hole_in_ones):
+    """when mask is all ones, image should be unmodified"""
+    mask = np.ones((5, 5))
+    result = fillnodata(hole_in_ones, mask)
+    assert (hole_in_ones == result).all()
diff --git a/tests/test_rio_calc.py b/tests/test_rio_calc.py
--- a/tests/test_rio_calc.py
+++ b/tests/test_rio_calc.py
@@ -121,7 +121,7 @@ def test_fillnodata(tmpdir):
         assert src.count == 3
         assert src.meta['dtype'] == 'uint8'
         data = src.read(masked=True)
-        assert round(data.mean(), 1) == 60.6
+        assert round(data.mean(), 1) == 58.6
 
 
 def test_fillnodata_map(tmpdir):
@@ -135,8 +135,8 @@ def test_fillnodata_map(tmpdir):
         assert src.count == 3
         assert src.meta['dtype'] == 'uint8'
         data = src.read(masked=True)
-        assert round(data.mean(), 1) == 60.6
-
+        assert round(data.mean(), 1) == 58.6
+        assert data[0][60][60] > 0
 
 def test_sieve_band(tmpdir):
     outfile = str(tmpdir.join('out.tif'))