Repository: rasterio/rasterio

Issue:
How to handle statistics
I like statistics in my rasters since they load nicely into various GIS software that I use, which includes ArcGIS. Without statistics (or the correct statistics) the rasters don't look correct in these software.

Up to now, I generally need the help of osgeo.gdal to add/update band statistics after rasterio processing, e.g.:

``` python
ds = gdal.Open(fname, gdal.GA_Update)
for i in range(ds.RasterCount):
    ds.GetRasterBand(i + 1).ComputeStatistics(0)
ds = band = None  # save, close
```

How could this be done with rasterio? A few ideas:
- Have optional arguments for updatable rasters with `rasterio.open()`, e.g. `stats=True` or `approx_stats=True`, which are used when processing `close()`.
- Add a method like `obj.calc_stats(approx=False)`, which can be manually called while the raster dataset is open. Should it return the four statistics that [ComputeStatistics](http://www.gdal.org/classGDALRasterBand.html#a48883c1dae195b21b37b51b10e910f9b) returns? Note that this data are also available through the metadata tags.
- Start a `rio edit [--stats] [--approx_stats]` tool, similar [to this enhancement to gdal_edit.py](http://trac.osgeo.org/gdal/ticket/5805).



Discussion:
@mwtoews I like the idea of a rio-edit command very much. Separating the editing of data and metadata could be nice, but I'm happy to leave that up to you.

In the library API: datasets have a update_tags() method that can be used to set metadata. Will this cover statistics?

I think I'm in favor of a `stats=True` keyword arg. The open function already takes and passes on dataset creation options (like `compress='LZW'` for GeoTIFFs) and stats on output is IMO in the same category.

GDAL typically stores the statistics in the metadata tags (with keys that start with "STATISTICS_"). See more at  [`GDALRasterBand::SetStatistics`](http://www.gdal.org/classGDALRasterBand.html#a1c60c45cb9535999a5318266cc429421). They aren't stored anywhere special, like nodata or geotransform are. Sure `update_tags()` could be used, but there are four of them, and I'm guessing that the internal `GetStatistics()` methods calculates them efficiently than a user would with NumPy.

Could a new method be added to `_io.RasterUpdater`? How about `calc_stats(approx=False)`, which returns a tuple `(min, max, avg, sd)`. Internally, it will just call `ComputeStatistics()`, which I think does the same as `GetStatistics()` then `SetStatistics()`. Alternative names would be possibly just `stats(approx=False)`, returning same 4 tuple.

And as for rio-edit, it would essentially be very similar to [gdal_edit.py](http://www.gdal.org/gdal_edit.html).

Was there any progress on this? Had a quick check, but couldn't see anything in current version of `rasterio`.

Having a wrapper for GDAL's [`ComputeStatistics()`](https://gdal.org/doxygen/classGDALRasterBand.html#a48883c1dae195b21b37b51b10e910f9b) method (e.g. `calc_stats(approx=False)` as discussed above) would be really handy as it would benefit from all the optimizations done within GDAL (just see [`statistics.txt`](https://github.com/OSGeo/gdal/blob/master/gdal/gcore/statistics.txt) for an exhaustive list!). 

Although calculating stats for small/medium rasters using `rasterio` is pretty trivial (just read in the full dataset and use `np.mean()` etc), doing so for larger rasters (which don't fit in memory) is more of a pain since you have to use windowed reads to process the raster block-by-block and manually accumulate the stats. Having the `calc_stats()` wrapper would be great in this case because GDAL already handles that.
@peetw no progress that I'm aware of, but this feature as discussed would be a welcome feature.
@peetw @mwtoews storing basic statistics in dataset metadata is great, I really get it. I'm a bit uncomfortable in that formats don't guarantee that data and statistics agree. If you download a GeoTIFF from the internet with pixel stats stored in the metadata, how do you know that the stats are correct? You have to compute them yourself and check. And then if you modify the pixels in the GeoTIFF (or a copy), the metadata doesn't get updated and the stored stats could be out of sync. There's no format-level spec for statistics.

The thing I've objected to in the past about ComputeStatistics is this: 

> Once computed, the statistics will generally be "set" back on the raster band using SetStatistics().

Writes to disk (like an .aux.xml file) as a side effect of a method that looks like it is idempotent still feels off to me.

Speaking of welcome features, what would you think about the project providing an API for applying ufuncs (or other functions) to very big datasets? One based on rio-calc's chunker (https://github.com/mapbox/rasterio/blob/master/rasterio/rio/calc.py#L36) maybe, or a better one.
For what it's worth, I think I have the minimal example of computing stats with dask in https://gist.github.com/sgillies/8d0b77699dea4de9ca1ab6ec5552e1c8. We won't depend on dask for stats, but it's a nice option for many applications.

Resolving patch:
diff --git a/rasterio/rio/info.py b/rasterio/rio/info.py
--- a/rasterio/rio/info.py
+++ b/rasterio/rio/info.py
@@ -3,6 +3,7 @@
 
 import json
 
+from attr import asdict
 import click
 
 import rasterio
@@ -105,10 +106,7 @@ def info(ctx, input, aspect, indent, namespace, meta_member, verbose, bidx,
             info['gcps']['transform'] = from_gcps(gcps)
 
         if verbose:
-            stats = [{'min': float(b.min()),
-                      'max': float(b.max()),
-                      'mean': float(b.mean())
-                      } for b in src.read(masked=masked)]
+            stats = [asdict(src.statistics(bidx)) for bidx in src.indexes]
             info['stats'] = stats
             info['checksum'] = [src.checksum(i) for i in src.indexes]
 
@@ -117,11 +115,8 @@ def info(ctx, input, aspect, indent, namespace, meta_member, verbose, bidx,
                 for name in src.subdatasets:
                     click.echo(name)
             elif meta_member == 'stats':
-                band = src.read(bidx, masked=masked)
-                click.echo('%f %f %f' % (
-                    float(band.min()),
-                    float(band.max()),
-                    float(band.mean())))
+                st = src.statistics(bidx)
+                click.echo("{st.min} {st.max} {st.mean} {st.std}".format(st=st))
             elif meta_member == 'checksum':
                 click.echo(str(src.checksum(bidx)))
             elif meta_member:


Test patch:
diff --git a/tests/test_dataset.py b/tests/test_dataset.py
--- a/tests/test_dataset.py
+++ b/tests/test_dataset.py
@@ -7,6 +7,7 @@
 except ImportError:
     from mock import MagicMock
 
+from numpy.testing import assert_almost_equal
 import pytest
 
 import rasterio
@@ -69,3 +70,13 @@ def test_dataset_readonly_attributes(path_rgb_byte_tif):
     with pytest.raises(NotImplementedError):
         with rasterio.open(path_rgb_byte_tif) as dataset:
             dataset.crs = "foo"
+
+
+def test_statistics(path_rgb_byte_tif):
+    """Compute, store, and return basic statistics."""
+    with rasterio.open(path_rgb_byte_tif) as dataset:
+        stats = dataset.statistics(1)
+        assert stats.min == 0
+        assert stats.max == 255
+        assert_almost_equal(stats.mean, 29.947726688477)
+        assert_almost_equal(stats.std, 52.340921626611)
diff --git a/tests/test_rio_info.py b/tests/test_rio_info.py
--- a/tests/test_rio_info.py
+++ b/tests/test_rio_info.py
@@ -153,8 +153,7 @@ def test_info_stats(runner):
         main_group, ['info', 'tests/data/RGB.byte.tif', '--tell-me-more'])
     assert result.exit_code == 0
     assert '"max": 255.0' in result.output
-    assert '"min": 1.0' in result.output
-    assert '"mean": 44.4344' in result.output
+    assert '"min": 0.0' in result.output
 
 
 def test_info_stats_only(runner):
@@ -162,7 +161,7 @@ def test_info_stats_only(runner):
         main_group,
         ['info', 'tests/data/RGB.byte.tif', '--stats', '--bidx', '2'])
     assert result.exit_code == 0
-    assert result.output.startswith('1.000000 255.000000 66.02')
+    assert result.output.startswith("0.0 255.0")
 
 
 def test_info_colorinterp(runner):
@@ -428,6 +427,7 @@ def test_info_aws_unsigned(runner):
 
 @requires_gdal32(reason="Unsigned Azure requests require GDAL ~= 3.2")
 @pytest.mark.network
+@pytest.mark.skip(reason="Undiagnosed problem accessing this file")
 def test_info_azure_unsigned(monkeypatch, runner):
     """Unsigned access to public dataset works"""
     monkeypatch.setenv('AZURE_NO_SIGN_REQUEST', 'YES')