Repository: rasterio/rasterio

Issue:
Decimated read result differs depending on GDAL version
```
$ rio insp tests/data/alpha_masked_values.tif
Rasterio 1.0a1 Interactive Inspector (Python 3.5.1)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> src.read(4, out_shape=(19, 19), masked=False)[-2:, 0:5]
array([[  0,   0,   0, 255,   0],
       [255, 255, 255, 255,   0]], dtype=uint8)
>>> rasterio.__version__
'1.0a1'
>>> rasterio.__gdal_version__
'1.11.5'
```

versus 

```
$ rio insp tests/data/alpha_masked_values.tif
Rasterio 1.0a1 Interactive Inspector (Python 3.5.1)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> src.read(4, out_shape=(19, 19), masked=False)[-2:, 0:5]
array([[  0,   0,  32,  64,   0],
       [255, 255, 255, 255,   0]], dtype=uint8)
>>> rasterio.__version__
'1.0a1'
>>> rasterio.__gdal_version__
'2.1.1'
```

I'll start a new branch with a similar test so we can put it through the travis build matrix.

cc @dnomadb @sgillies 



Discussion:
A few things worth noting:
- This is attempting to read the alpha band (4)
- The alpha band definitely contains values other than 0 and 255
- The non-zero values of the alpha band are masking some otherwise valid RGB values
- Uses JPEG compression

The change in behavior happened between GDAL 1.11 and GDAL 2.0.  Not sure which is "right" but there's clearly a difference that we need to resolve.

![screen shot 2016-09-19 at 12 20 18 pm](https://cloud.githubusercontent.com/assets/1151287/18639856/959dfea4-7e63-11e6-8708-06857b9686b6.png)

Might be related to: https://github.com/mapbox/rasterio/issues/679#issuecomment-215981252 ?

Worked up a C example (rough) just to rule out any Rasterio weirdness. 

https://gist.github.com/perrygeo/ee3fcd8ce571003d32efa6552d572dbe

It's occurring in the C API too which indicates that there was some breaking change to `GDALRasterIO` in v2

Looking through the [gdal changes](https://trac.osgeo.org/gdal/wiki/Release/2.0.0-News) the only thing I could see is the addition of [RFC 51](https://trac.osgeo.org/gdal/wiki/rfc51_rasterio_resampling_progress) which adds resampling methods for decimated reads. But I haven't seen anything in there that explicitly calls out a backwards-incompatible change in the defaults.

@perrygeo i can neither confirm or deny because that particular read crashes my python 3.5/gdal 2.1.0 combo:

```
>>> src.read(4, out_shape=(19, 19), masked=False)[-2:, 0:5]
Fatal Python error: PyThreadState_Get: no current thread
Segmentation fault: 11
```

Not sure what's up with that.

Can confirm segfault with GDAL 2.1.0. 

GDAL 2.1.1 doesn't segfault (but shows this different behavior). May be related to this fix: https://trac.osgeo.org/gdal/ticket/6493 which landed post 2.1.0

Setting `GTIFF_IMPLICIT_JPEG_OVR=NO` 
- cures the segfault on `gdal>=2.0,<2.1.1`
- makes the behavior consistent with results from `gdal<2.0`

So it would appear that turning off implicit JPEG overviews is essential to avoiding segfaults and keeping behaviour consistent across gdal versions. Not sure what this means for Rasterio and rio plugins yet...

Trying with Env on travis right now.

💚  from travis. Here's the test.

```
def test_decimated_read_values():
    # These are the values expected from GDAL 1.11
    expected = np.array([
        [  0,   0,   0, 255,   0],
        [255, 255, 255, 255,   0]], dtype='uint8')

    # This is the only way currently to ensure consistent and non-crashy behavior
    # across all GDAL versions
    with rasterio.Env(GTIFF_IMPLICIT_JPEG_OVR=False):
        with rasterio.open('tests/data/alpha_masked_values.tif') as src:
            result = src.read(4, out_shape=(19, 19), masked=False)[-2:, 0:5]

    assert np.array_equal(expected, result)
```

Still pondering what this means for Rasterio. Should we turn it off by default? Just document it? File a bug report in the GDAL tracker?

One data point to consider is that turning implicit jpeg overviews off resulted in reads that were ~3x slower.

@perrygeo let's make `GTIFF_IMPLICIT_JPEG_OVR=NO` a default in `env.py`. Seems like the least surprising thing, and we can turn it on when we think we need it.


Resolving patch:
diff --git a/rasterio/env.py b/rasterio/env.py
--- a/rasterio/env.py
+++ b/rasterio/env.py
@@ -18,7 +18,8 @@
 
 # Rasterio defaults
 default_options = {
-    'CHECK_WITH_INVERT_PROJ': True
+    'CHECK_WITH_INVERT_PROJ': True,
+    'GTIFF_IMPLICIT_JPEG_OVR': False
 }
 
 class Env(object):


Test patch: