Repository: rasterio/rasterio

Issue:
read_masks() return only 255s when using `boundless=True`  
This might be linked to  #1191 


## Expected behavior and actual behavior.
should return expected mask when setting `boundless=True` 

## Steps to reproduce the problem.


```python 
f = 'http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif'

with rasterio.open(f) as src:
    out_shape = (src.count, 512, 512)
    window = Window(48000, 0, 1024, 1024) # Reading part of a file with mask values
    # Read the data corresponding to the window
    data = src.read(window=window, out_shape=out_shape, boundless=True, resampling=Resampling.bilinear)
   
    # Read internal mask and resample to out_shape
    mask = src.read_masks(1, window=window, boundless=True, out_shape=(512, 512), resampling=Resampling.bilinear)
    # This return only 255s value 

    # Read internal mask boundless
    mask_no_boundless = src.read_masks(1, window=window, out_shape=(512, 512), resampling=Resampling.bilinear)
    # This return the expected mask
```
full example: https://gist.github.com/vincentsarago/2e6eacc25d957490c99ecb1496226378


## Operating system

Mac OS X 10

## Rasterio version and provenance

`rasterio-1.0a12-cp36-cp36m-macosx_10_9_intel.macosx_10_9_x86_64.whl` 

cc @mojodna 


Discussion:
@vincentsarago definitely sounds linked, as I seem to remember boundless reads going through a `WarpedVRT`.
Confirmed. ~We're not yet creating a mask band for the VRT, that's the problem.~
More details.

The GeoTIFF gets its mask from a .msk sidecar file. 

```
Files: /vsicurl/http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif
       /vsicurl/http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif.msk
```

Calling `GDALCreateWarpedVRT` doesn't seem to pick up the sidecar. I don't know whether this is intended or not and will ask for help on gdal-dev.
I asked my question poorly on gdal-dev, but was assured that `GDALAutoCreateWarpedVRT` does the right thing. Rasterio's `WarpedVRT` does not, even when it calls `GDALAutoCreateWarpedVRT`, which leads me to believe that `create_warp_options` is setting some improper warp options that are interfering with mask file detection.
@sgillies As I discovered in #1283, providing `dst_height`, `dst_width`, and `dst_transform` triggers a call to [`GDALCreateWarpedVRT()`](https://github.com/mapbox/rasterio/blob/2ecbf348ca775ddb0ed43d0712633b3c52a8aad5/rasterio/_warp.pyx#L753), *not* `GDALAutoCreateWarpedVRT()`, which must not be picking up the `PER_DATASET` mask band.

Based on the [VRT docs](http://www.gdal.org/gdal_vrttut.html) I would expect to see a `<MaskBand>` element in the VRT, however neither the VRT produced by `GDALAutoCreateWarpedVRT()` or `GDALCreateWarpedVRT()` seem to add it.

```
<MaskBand>
  <VRTRasterBand dataType="Byte">
    <SimpleSource>
      <SourceFilename relativeToVRT="1">utm.tif</SourceFilename>
      <SourceBand>mask,1</SourceBand>
      <SrcRect xOff="0" yOff="0" xSize="512" ySize="512"/>
      <DstRect xOff="0" yOff="0" xSize="512" ySize="512"/>
    </SimpleSource>
  </VRTRasterBand>
</MaskBand>
```

The [VRT driver implements the `CreateMaskBand()`](http://www.gdal.org/classVRTDataset.html#aab8ad2b079703a9f5336ee40151cc144) method, but always creates a sidecar mask file and requires mask flags.  This is something the VRT should be inheriting from the parent dataset.
I wasn't able to find anything explictly about masks in the `GDALAutoCreateWarpedVRT` code and am thinking that it must be a side effect of nodata values in the warp operations struct.
@sgillies Done editing.

I think you're right.  As [you mentioned on `gdal-dev`](https://lists.osgeo.org/pipermail/gdal-dev/2018-February/048159.html), GDAL is indicating this dataset has a dataset-level mask and is picking up the mask sidecar file:

```
$ gdalinfo /vsicurl/http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif | grep "Mask Flags"
  Mask Flags: PER_DATASET 
  Mask Flags: PER_DATASET 
  Mask Flags: PER_DATASET
```

Downloading the image without the sidecar mask file and probing again shows that the image has no mask flags, meaning both an implicit `ALL_VALID` and that the image lacks a `nodata` value.

So, I think the only reason this works:

```
    # Read internal mask boundless
    mask_no_boundless = src.read_masks(1, window=window, out_shape=(512, 512), resampling=Resampling.bilinear)
    # This return the expected mask
```

is because `GDALWarpOptions`'s `padfSrcNoDataReal` value is [defaulting to `0`](https://github.com/mapbox/rasterio/blob/2ecbf348ca775ddb0ed43d0712633b3c52a8aad5/rasterio/_warp.pyx#L112) when doing a [boundless read on a mask band](https://github.com/mapbox/rasterio/blob/2ecbf348ca775ddb0ed43d0712633b3c52a8aad5/rasterio/_io.pyx#L526), and that happens to be a valid `nodata` value for this image.

It seems like neither `GDALCreateWarpedVRT()` or `GDALAutoCreateWarpedVRT()` are adding a [`<MaskBand>`](http://www.gdal.org/gdal_vrttut.html) element.  There is a [`VRTDataset::CreateMaskBand()`](http://www.gdal.org/classVRTDataset.html#aab8ad2b079703a9f5336ee40151cc144) method, but it always creates a sidecar `.msk` file.

I'm sure there are similar lurking issues with boundless reads in `src.read(boundless=True, masked=True)` given that it does [almost the](https://github.com/mapbox/rasterio/blob/2ecbf348ca775ddb0ed43d0712633b3c52a8aad5/rasterio/_io.pyx#L369) exact [same thing](https://github.com/mapbox/rasterio/blob/2ecbf348ca775ddb0ed43d0712633b3c52a8aad5/rasterio/_io.pyx#L387).
Also this duplicates #1266.
Running:

```
$ gdalbuildvrt test.vrt /vsicurl/http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif
```

produces a VRT containing a `<MaskBand>` element:

```
  <MaskBand>
    <VRTRasterBand dataType="Byte">
      <SimpleSource>
        <SourceFilename relativeToVRT="0">/vsicurl/http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif</SourceFilename>
        <SourceBand>mask,1</SourceBand>
        <SourceProperties RasterXSize="48948" RasterYSize="15632" DataType="Byte" BlockXSize="512" BlockYSize="512" />
        <SrcRect xOff="0" yOff="0" xSize="48948" ySize="15632" />
        <DstRect xOff="0" yOff="0" xSize="48948" ySize="15632" />
      </SimpleSource>
    </VRTRasterBand>
  </MaskBand>
```

but `$ gdalinfo test.vrt` produces all kinds of errors when reporting band information.  Regardless, it can probably [provide some prior art](https://github.com/OSGeo/gdal/blob/1c27697af2e35ce8ce1b87c7d2bae7bb079a3122/gdal/apps/gdalbuildvrt_lib.cpp#L900).
@geowurster Would have been appropriate to report the issue you mentionned in above comment in GDAL Trac. Anyway, fixed per https://trac.osgeo.org/gdal/changeset/41752 + https://trac.osgeo.org/gdal/changeset/41753
@geowurster thanks for looking into this! I agree that gdalbuildvrt is the place to look for examples.
I don't think this is only true for dataset with sidecar mask. 

Another observation, while checking the http calls to monitor the byte transferred I noticed that when we have `window` and `boundless=True` (meaning when we use `WrapVRT`) we don't do any http request but I cannot understand why.

### Just opening the dataset
```
GDAL_CACHEMAX=0 CPL_CURL_VERBOSE=YES python -c 'import rasterio; src = rasterio.open("https://s3-us-west-2.amazonaws.com/remotepixel-us-west-2/data/tile_ycbcr.tif");' 2>&1 > /dev/null | grep 'Content-Length:'
< Content-Length: 647164
< Content-Length: 49152
```

### Opening and read full mask
```
$ GDAL_CACHEMAX=0 CPL_CURL_VERBOSE=YES python -c 'import rasterio; src = rasterio.open("https://s3-us-west-2.amazonaws.com/remotepixel-us-west-2/data/tile_ycbcr.tif");  src.read_masks(1);' 2>&1 > /dev/null | grep 'Content-Length:'
< Content-Length: 647164
< Content-Length: 49152
< Content-Length: 24572
```

### Opening and read partial mask
```
 $ GDAL_CACHEMAX=0 CPL_CURL_VERBOSE=YES python -c 'import rasterio; src = rasterio.open("https://s3-us-west-2.amazonaws.com/remotepixel-us-west-2/data/tile_ycbcr.tif");  src.read_masks(1, boundless=True, window=(0,0,10,10));' 2>&1 > /dev/null | grep 'Content-Length:'
< Content-Length: 647164
< Content-Length: 49152
```

cc @mojodna @sgillies @geowurster 
I sent an email to gdal-dev to find out whether there's any possibility of a source raster's mask surfacing in a warped VRT. At the moment, it seems like there is not, except when the source's mask is derived from an alpha band. Otherwise, it looks like we have to materialize the VRT, read data and then derive a mask based on nodata values.
@vincentsarago I've got the fix for this in #1361. In more discussion on gdal-dev, Even pointed out that specifying use of a destination alpha band for the VRT, as in `gdalwarp -dstalpha`, is the solution. This is what we'll do when the source dataset has a mask and no nodata value.

Resolving patch:
diff --git a/rasterio/rio/insp.py b/rasterio/rio/insp.py
--- a/rasterio/rio/insp.py
+++ b/rasterio/rio/insp.py
@@ -28,8 +28,6 @@
     plt = None
 
 
-logger = logging.getLogger('rasterio')
-
 Stats = collections.namedtuple('Stats', ['min', 'max', 'mean'])
 
 # Collect dictionary of functions for use in the interpreter in main()
@@ -73,7 +71,7 @@ def main(banner, dataset, alt_interpreter=None):
 @click.pass_context
 def insp(ctx, input, mode, interpreter):
     """Open the input file in a Python interpreter."""
-    logger = logging.getLogger('rio')
+    logger = logging.getLogger()
     try:
         with ctx.obj['env']:
             with rasterio.open(input, mode) as src:
diff --git a/rasterio/vrt.py b/rasterio/vrt.py
--- a/rasterio/vrt.py
+++ b/rasterio/vrt.py
@@ -1,8 +1,14 @@
 """rasterio.vrt: a module concerned with GDAL VRTs"""
 
+import xml.etree.ElementTree as ET
+
+import rasterio
 from rasterio._warp import WarpedVRTReaderBase
-from rasterio.windows import WindowMethodsMixin
+from rasterio.dtypes import _gdal_typename
+from rasterio.enums import MaskFlags
+from rasterio.path import parse_path, vsi_path
 from rasterio.transform import TransformMethodsMixin
+from rasterio.windows import WindowMethodsMixin
 
 
 class WarpedVRT(WarpedVRTReaderBase, WindowMethodsMixin,
@@ -61,3 +67,92 @@ def __del__(self):
 
     def close(self):
         self.stop()
+
+
+def _boundless_vrt_doc(src_dataset, nodata=None, width=None, height=None, transform=None):
+    """Make a VRT XML document."""
+
+    nodata = nodata or src_dataset.nodata
+    width = width or src_dataset.width
+    height = height or src_dataset.height
+    transform = transform or src_dataset.transform
+
+    vrtdataset = ET.Element('VRTDataset')
+    vrtdataset.attrib['rasterYSize'] = str(height)
+    vrtdataset.attrib['rasterXSize'] = str(width)
+    srs = ET.SubElement(vrtdataset, 'SRS')
+    srs.text = src_dataset.crs.wkt
+    geotransform = ET.SubElement(vrtdataset, 'GeoTransform')
+    geotransform.text = ','.join([str(v) for v in transform.to_gdal()])
+
+    for bidx, ci, block_shape, dtype in zip(src_dataset.indexes, src_dataset.colorinterp, src_dataset.block_shapes, src_dataset.dtypes):
+        vrtrasterband = ET.SubElement(vrtdataset, 'VRTRasterBand')
+        vrtrasterband.attrib['dataType'] = _gdal_typename(dtype)
+        vrtrasterband.attrib['band'] = str(bidx)
+
+        if nodata is not None:
+            nodatavalue = ET.SubElement(vrtrasterband, 'NoDataValue')
+            nodatavalue.text = str(nodata)
+
+        colorinterp = ET.SubElement(vrtrasterband, 'ColorInterp')
+        colorinterp.text = ci.name.capitalize()
+
+        simplesource = ET.SubElement(vrtrasterband, 'SimpleSource')
+
+        sourcefilename = ET.SubElement(simplesource, 'SourceFilename')
+        sourcefilename.attrib['relativeToVRT'] = "0"
+        sourcefilename.text = vsi_path(parse_path(src_dataset.name))
+
+        sourceband = ET.SubElement(simplesource, 'SourceBand')
+        sourceband.text = str(bidx)
+        sourceproperties = ET.SubElement(simplesource, 'SourceProperties')
+        sourceproperties.attrib['RasterXSize'] = str(width)
+        sourceproperties.attrib['RasterYSize'] = str(height)
+        sourceproperties.attrib['dataType'] = _gdal_typename(dtype)
+        sourceproperties.attrib['BlockYSize'] = str(block_shape[0])
+        sourceproperties.attrib['BlockXSize'] = str(block_shape[1])
+        srcrect = ET.SubElement(simplesource, 'SrcRect')
+        srcrect.attrib['xOff'] = '0'
+        srcrect.attrib['yOff'] = '0'
+        srcrect.attrib['xSize'] = str(src_dataset.width)
+        srcrect.attrib['ySize'] = str(src_dataset.height)
+        dstrect = ET.SubElement(simplesource, 'DstRect')
+        dstrect.attrib['xOff'] = str((src_dataset.transform.xoff - transform.xoff) / transform.a)
+        dstrect.attrib['yOff'] = str((src_dataset.transform.yoff - transform.yoff) / transform.e)
+        dstrect.attrib['xSize'] = str(src_dataset.width)
+        dstrect.attrib['ySize'] = str(src_dataset.height)
+
+        if src_dataset.nodata is not None:
+            nodata_elem = ET.SubElement(simplesource, 'NODATA')
+            nodata_elem.text = str(src_dataset.nodata)
+
+    if all(MaskFlags.per_dataset in flags for flags in src_dataset.mask_flag_enums):
+        maskband = ET.SubElement(vrtdataset, 'MaskBand')
+        vrtrasterband = ET.SubElement(maskband, 'VRTRasterBand')
+        vrtrasterband.attrib['dataType'] = 'Byte'
+
+        simplesource = ET.SubElement(vrtrasterband, 'SimpleSource')
+        sourcefilename = ET.SubElement(simplesource, 'SourceFilename')
+        sourcefilename.attrib['relativeToVRT'] = "0"
+        sourcefilename.text = vsi_path(parse_path(src_dataset.name))
+
+        sourceband = ET.SubElement(simplesource, 'SourceBand')
+        sourceband.text = 'mask,1'
+        sourceproperties = ET.SubElement(simplesource, 'SourceProperties')
+        sourceproperties.attrib['RasterXSize'] = str(width)
+        sourceproperties.attrib['RasterYSize'] = str(height)
+        sourceproperties.attrib['dataType'] = 'Byte'
+        sourceproperties.attrib['BlockYSize'] = str(block_shape[0])
+        sourceproperties.attrib['BlockXSize'] = str(block_shape[1])
+        srcrect = ET.SubElement(simplesource, 'SrcRect')
+        srcrect.attrib['xOff'] = '0'
+        srcrect.attrib['yOff'] = '0'
+        srcrect.attrib['xSize'] = str(src_dataset.width)
+        srcrect.attrib['ySize'] = str(src_dataset.height)
+        dstrect = ET.SubElement(simplesource, 'DstRect')
+        dstrect.attrib['xOff'] = str((src_dataset.transform.xoff - transform.xoff) / transform.a)
+        dstrect.attrib['yOff'] = str((src_dataset.transform.yoff - transform.yoff) / transform.e)
+        dstrect.attrib['xSize'] = str(src_dataset.width)
+        dstrect.attrib['ySize'] = str(src_dataset.height)
+
+    return ET.tostring(vrtdataset)


Test patch:
diff --git a/tests/test_boundless_read.py b/tests/test_boundless_read.py
--- a/tests/test_boundless_read.py
+++ b/tests/test_boundless_read.py
@@ -7,7 +7,10 @@
 import rasterio
 from rasterio.windows import Window
 
+from .conftest import requires_gdal21
 
+
+@requires_gdal21(reason="Pixel equality tests require float windows and GDAL 2.1")
 @given(col_start=st.integers(min_value=-700, max_value=0),
        row_start=st.integers(min_value=-700, max_value=0),
        col_stop=st.integers(min_value=0, max_value=700),
diff --git a/tests/test_vrt.py b/tests/test_vrt.py
new file mode 100644
--- /dev/null
+++ b/tests/test_vrt.py
@@ -0,0 +1,24 @@
+"""Tests of the rasterio.vrt module"""
+
+import rasterio
+import rasterio.vrt
+
+
+def test_boundless_vrt(path_rgb_byte_tif):
+    with rasterio.open(path_rgb_byte_tif) as rgb:
+        doc = rasterio.vrt._boundless_vrt_doc(rgb)
+        assert doc.startswith(b'<VRTDataset')
+        with rasterio.open(doc.decode('ascii')) as vrt:
+            assert rgb.count == vrt.count
+            assert rgb.dtypes == vrt.dtypes
+            assert rgb.mask_flag_enums == vrt.mask_flag_enums
+
+
+def test_boundless_msk_vrt(path_rgb_msk_byte_tif):
+    with rasterio.open(path_rgb_msk_byte_tif) as msk:
+        doc = rasterio.vrt._boundless_vrt_doc(msk)
+        assert doc.startswith(b'<VRTDataset')
+        with rasterio.open(doc.decode('ascii')) as vrt:
+            assert msk.count == vrt.count
+            assert msk.dtypes == vrt.dtypes
+            assert msk.mask_flag_enums == vrt.mask_flag_enums
diff --git a/tests/test_warpedvrt.py b/tests/test_warpedvrt.py
--- a/tests/test_warpedvrt.py
+++ b/tests/test_warpedvrt.py
@@ -7,7 +7,7 @@
 import boto3
 import pytest
 
-from .conftest import requires_gdal21
+from .conftest import requires_gdal21, requires_gdal2
 import rasterio
 from rasterio.crs import CRS
 from rasterio.enums import Resampling, MaskFlags, ColorInterp
@@ -76,7 +76,7 @@ def test_warped_vrt_dst_alpha(path_rgb_byte_tif):
 def test_warped_vrt_msk_default(path_rgb_msk_byte_tif):
     """Add an alpha band to the VRT to access per-dataset mask of a source"""
     with rasterio.open(path_rgb_msk_byte_tif) as src:
-        vrt = WarpedVRT(src, crs=DST_CRS)
+        vrt = WarpedVRT(src, crs=DST_CRS, add_alpha=True)
         assert vrt.src_nodata is None
         assert vrt.dst_nodata is None
         assert vrt.count == 4
@@ -247,19 +247,18 @@ def test_crs_should_be_set(path_rgb_byte_tif, tmpdir, complex):
             assert src.crs
 
 
-def test_add_alpha_read(path_rgb_byte_tif):
-    """Boundless read of a VRT with added alpha succeeds"""
-    with rasterio.open(path_rgb_byte_tif) as src, WarpedVRT(src, add_alpha=True) as vrt:
-        assert vrt.count == 4
-        assert vrt.colorinterp[3] == ColorInterp.alpha
-        data = vrt.read(boundless=True, window=Window(-200, -200, 1000, 1000), out_shape=((3, 600, 600)))
+def test_boundless_read_prohibited(path_rgb_byte_tif):
+    """Boundless read of a VRT is prohibited"""
+    with rasterio.open(path_rgb_byte_tif) as src, WarpedVRT(src) as vrt:
+        with pytest.raises(ValueError):
+            vrt.read(boundless=True, window=Window(-200, -200, 1000, 1000), out_shape=((3, 600, 600)))
 
 
-def test_default_add_alpha_read(path_rgb_msk_byte_tif):
-    """An alpha band is added if add_alpha is unspecified and source has .msk"""
-    with rasterio.open(path_rgb_msk_byte_tif) as src, WarpedVRT(src) as vrt:
-        assert vrt.count == 4
-        assert vrt.colorinterp[3] == ColorInterp.alpha
+def test_boundless_masks_read_prohibited(path_rgb_byte_tif):
+    """Boundless masks read of a VRT is prohibited"""
+    with rasterio.open(path_rgb_byte_tif) as src, WarpedVRT(src) as vrt:
+        with pytest.raises(ValueError):
+            vrt.read_masks(boundless=True, window=Window(-200, -200, 1000, 1000), out_shape=((3, 600, 600)))
 
 
 def test_no_add_alpha_read(path_rgb_msk_byte_tif):