Repository: cogeotiff/rio-cogeo

Issue:
Note about Opinions! 
While rio-cogeo follows the COG specification, it also make some choice. By default we decided to enforce internal tilings (tiling is optional for files that are < 1024x1024) and overview (those are optional).

I personally thinks those two are important but the user still have the possibility to change the default by using options (in CLI). 

Let's add a note on the README about the choice made for the user.

Discussion:

Patch:
diff --git a/CHANGES.txt b/CHANGES.txt
index 46e78c2..22b6258 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,3 +1,12 @@
+
+1.0b3 (2019-03-30)
+------------------
+
+Breacking Changes:
+
+- remove deprecated YCBCR profile
+- 512x512 dataset without internal tiling are valid
+
 1.0b2 (2019-03-27)
 ------------------
 
@@ -11,7 +20,7 @@ Breacking Changes:
 
 Breacking Changes:
 
-- refactor utils.get_maximum_overview_level to get rasterio dataset 
+- refactor utils.get_maximum_overview_level to get rasterio dataset
 as input and reduce the number of dataset opennings (#61)
 
 
diff --git a/README.rst b/README.rst
index 72868b9..d523525 100644
--- a/README.rst
+++ b/README.rst
@@ -14,6 +14,18 @@ Cloud Optimized GeoTIFF (COG) creation and validation plugin for Rasterio
    :target: https://codecov.io/gh/cogeotiff/rio-cogeo
 
 
+Cloud Optimized GeoTIFF
+=======================
+
+This plugin aim to facilitate the creation and validation of Cloud Optimized
+GeoTIFF (COG or COGEO). While it respects the
+`COG specifications <https://github.com/cogeotiff/cog-spec/blob/master/spec.md>`__, this plugin also
+enforce several features:
+
+- **Internal overviews** (User can remove overview with option `--overview-level 0`)
+- **Internal tiles** (default profiles have 512x512 internal tiles)
+
+
 Install
 =======
 
@@ -31,6 +43,7 @@ Or install from source:
    $ pip install -U pip
    $ pip install -e .
 
+
 Usage
 =====
 
@@ -59,7 +72,7 @@ Usage
 
   Options:
     -b, --bidx BIDX                 Band indexes to copy.
-    -p, --cog-profile [ycbcr|jpeg|webp|zstd|lzw|deflate|packbits|raw] CloudOptimized GeoTIFF profile (default: deflate).
+    -p, --cog-profile [jpeg|webp|zstd|lzw|deflate|packbits|raw] CloudOptimized GeoTIFF profile (default: deflate).
     --nodata NUMBER|nan             Set nodata masking values for input dataset.
     --add-mask                      Force output dataset creation with an internal mask (convert alpha band or nodata to mask).
     --overview-level INTEGER        Overview level (if not provided, appropriate overview level will be selected until the
@@ -96,26 +109,12 @@ Examples
   $ rio cogeo validate mydataset_jpeg.tif
 
   # Create a COGEO with JPEG profile and the first 3 bands of the data and add internal mask
-  $ rio cogeo create mydataset.tif mydataset_jpeg.tif -b 1,2,3 --add-mask --cog-profile JPEG
-
-  # Create a COGEO without compression and with 1024x1024 block size and 256 overview blocksize
-  $ rio cogeo create mydataset.tif mydataset_raw.tif --co BLOCKXSIZE=1024 --co BLOCKYSIZE=1024 --cog-profile raw --overview-blocksize 256
-  $ GDAL_TIFF_OVR_BLOCKSIZE=256 rio cogeo create mydataset.tif mydataset_raw.tif --co BLOCKXSIZE=1024 --co BLOCKYSIZE=1024 --cog-profile raw
+  $ rio cogeo create mydataset.tif mydataset_jpeg.tif -b 1,2,3 --add-mask --cog-profile jpeg
 
 
 Default COGEO profiles
 ======================
 
-Profiles can be extended by providing '--co' option in command line
-(e.g: rio cogeo create mydataset.tif mydataset_zstd.tif -b 1,2,3 --profile deflate --co "COMPRESS=ZSTD" )
-
-**YCbCr** *DEPRECATED in 1.0*
-
-- JPEG compression
-- PIXEL interleave
-- YCbCr colorspace
-- limited to uint8 datatype and 3 bands data
-
 **JPEG**
 
 - JPEG compression
@@ -160,13 +159,28 @@ Profiles can be extended by providing '--co' option in command line
 
 Default profiles are tiled with 512x512 blocksizes.
 
+Profiles can be extended by providing '--co' option in command line
+
+.. code-block:: console
+
+    # Create a COGEO without compression and with 1024x1024 block size and 256 overview blocksize
+    $ rio cogeo create mydataset.tif mydataset_raw.tif --co BLOCKXSIZE=1024 --co BLOCKYSIZE=1024 --cog-profile raw --overview-blocksize 256
+
+
 Overview levels
 ===============
 
 By default rio cogeo will calculate the optimal overview level based on dataset
 size and internal tile size (overview should not be smaller than internal tile
 size (e.g 512px). Overview level will be translated to decimation level of
-power of two.
+power of two:
+
+.. code-block:: python
+
+  overview_level = 3
+  overviews = [2 ** j for j in range(1, overview_level + 1)]
+  print(overviews)
+  [2, 4, 8]
 
 Internal tile size
 ==================
@@ -177,14 +191,30 @@ This can be updated by passing `--co BLOCKXSIZE=64 --co BLOCKYSIZE=64` options.
 **Web tiling optimization**
 
 if the input dataset is aligned to web mercator grid, the internal tile size
-should be equal to the web map tile size (256 or 512px) output dataset is
-compressed,
+should be equal to the web map tile size (256 or 512px). Dataset should be compressed.
 
 if the input dataset is not aligned to web mercator grid, the tiler will need
 to fetch multiple internal tiles. Because GDAL can merge range request, using
 small internal tiles (e.g 128) will reduce the number of byte transfered and
 minimized the useless bytes transfered.
 
+
+GDAL configuration to merge consecutive range requests
+
+.. code-block:: console
+
+    GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES
+    GDAL_HTTP_MULTIPLEX=YES
+    GDAL_HTTP_VERSION=2
+
+
+GDAL Version
+============
+
+It is recommanded to use GDAL > 2.3.2. Previous version might not be able to
+create proper COGs (ref: https://github.com/OSGeo/gdal/issues/754).
+
+
 Nodata, Alpha and Mask
 ======================
 
@@ -209,6 +239,30 @@ Using internal nodata value with lossy compression (`webp`, `jpeg`) is not
 recommanded. Please use internal masking (or alpha band if using webp).
 
 
+Statistics
+==========
+
+Some libraries might request to use COGs with statistics written in the internal
+metadata. **rio-cogeo** doesn't calculate nor copy those when creating the output
+dataset (because statistics may change due to lossy compression).
+To add the statistics to the output dataset you could use the code above:
+
+.. code-block:: python
+
+  import rasterio
+
+  with rasterio.open("my-data.tif", "r+") as src_dst:
+      for b in src_dst.indexes:
+          band = src_dst.read(indexes=b, masked=masked)
+          stats = {
+              'min': float(band.min()),
+              'max': float(band.max()),
+              'mean': float(band.mean())
+              'stddev': float(band.std())
+          }
+          src_dst.update_tags(b, **stats)
+
+
 Contribution & Development
 ==========================
 
@@ -239,4 +293,4 @@ Extras
 
 Blog post on good and bad COG formats: https://medium.com/@_VincentS_/do-you-really-want-people-using-your-data-ec94cd94dc3f
 
-Checkout **rio-glui** (https://github.com/mapbox/rio-glui/) rasterio plugin to explore COG locally in your web browser.
+Checkout `**rio-glui** <https://github.com/mapbox/rio-glui/>__` rasterio plugin to explore COG locally in your web browser.
diff --git a/rio_cogeo/__init__.py b/rio_cogeo/__init__.py
index 03aa76e..d3f2b36 100644
--- a/rio_cogeo/__init__.py
+++ b/rio_cogeo/__init__.py
@@ -1,3 +1,3 @@
 """rio_cogeo"""
 
-__version__ = "1.0b2"
+__version__ = "1.0b3"
diff --git a/rio_cogeo/cogeo.py b/rio_cogeo/cogeo.py
index 6339a8d..9da270d 100644
--- a/rio_cogeo/cogeo.py
+++ b/rio_cogeo/cogeo.py
@@ -184,13 +184,13 @@ def cog_validate(src_path):
                     "Overviews found in external .ovr file. They should be internal"
                 )
 
-            if src.width >= 512 or src.height >= 512:
+            overviews = src.overviews(1)
+            if src.width > 512 or src.height > 512:
                 if not src.is_tiled:
                     errors.append(
                         "The file is greater than 512xH or 512xW, but is not tiled"
                     )
 
-                overviews = src.overviews(1)
                 if not overviews:
                     warnings.append(
                         "The file is greater than 512xH or 512xW, it is recommended "
@@ -208,7 +208,7 @@ def cog_validate(src_path):
             details["ifd_offsets"] = {}
             details["ifd_offsets"]["main"] = ifd_offset
 
-            if not overviews == sorted(overviews):
+            if overviews and overviews != sorted(overviews):
                 errors.append("Overviews should be sorted")
 
             for ix, dec in enumerate(overviews):
diff --git a/rio_cogeo/profiles.py b/rio_cogeo/profiles.py
index 02a4e03..8fb1bae 100644
--- a/rio_cogeo/profiles.py
+++ b/rio_cogeo/profiles.py
@@ -1,24 +1,8 @@
 """rio_cogeo.profiles: CloudOptimized profiles."""
 
-import warnings
-from rio_cogeo.errors import DeprecationWarning
 from rasterio.profiles import Profile
 
 
-class YCbCrProfile(Profile):
-    """Tiled, pixel-interleaved, JPEG-compressed, YCbCr colorspace, 8-bit GTiff."""
-
-    defaults = {
-        "driver": "GTiff",
-        "interleave": "pixel",
-        "tiled": True,
-        "blockxsize": 512,
-        "blockysize": 512,
-        "compress": "JPEG",
-        "photometric": "YCbCr",
-    }
-
-
 class JPEGProfile(Profile):
     """Tiled, pixel-interleaved, JPEG-compressed, YCbCr colorspace, 8-bit GTiff."""
 
@@ -120,7 +104,6 @@ class COGProfiles(dict):
         """Initialize COGProfiles dict."""
         self.update(
             {
-                "ycbcr": YCbCrProfile(),
                 "jpeg": JPEGProfile(),
                 "webp": WEBPProfile(),
                 "zstd": ZSTDProfile(),
@@ -133,12 +116,6 @@ class COGProfiles(dict):
 
     def get(self, key):
         """Like normal item access but error."""
-        if key == "ycbcr":
-            warnings.warn(
-                "'ycbcr' profile will be removed in 1.0, use 'jpeg' instead",
-                DeprecationWarning,
-            )
-
         if key not in (self.keys()):
             raise KeyError("{} is not a valid COG profile name".format(key))
 
diff --git a/setup.py b/setup.py
index 5e2f623..e2c5cdd 100644
--- a/setup.py
+++ b/setup.py
@@ -17,8 +17,8 @@ with open("README.rst") as f:
 inst_reqs = ["click", "rasterio[s3]>=1.0.9", "numpy~=1.15"]
 
 extra_reqs = {
-    "test": ["mock", "pytest", "pytest-cov"],
-    "dev": ["mock", "pytest", "pytest-cov", "pre-commit"],
+    "test": ["pytest", "pytest-cov"],
+    "dev": ["pytest", "pytest-cov", "pre-commit"],
 }
 
 setup(
diff --git a/tox.ini b/tox.ini
index 05b1dc0..c315ff8 100644
--- a/tox.ini
+++ b/tox.ini
@@ -27,3 +27,8 @@ deps =
 commands =
     {[testenv:build]commands}
     twine upload --skip-existing dist/*
+
+# Linter
+[flake8]
+exclude = .git,__pycache__,docs/source/conf.py,old,build,dist
+max-line-length = 90


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 6ab65b1..0c00104 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -20,8 +20,8 @@ raster_path_nodata = os.path.join(
 raster_path_missingnodata = os.path.join(
     os.path.dirname(__file__), "fixtures", "image_missing_nodata.tif"
 )
-raster_jpeg = os.path.join(
-    os.path.dirname(__file__), "fixtures", "validate", "nontiff.jpg"
+raster_invalid_cog = os.path.join(
+    os.path.dirname(__file__), "fixtures", "validate", "image_dec.tif"
 )
 
 
@@ -397,6 +397,19 @@ def test_cogeo_validNodataCustom():
 def test_cogeo_validate():
     """Should work as expected."""
     runner = CliRunner()
-    result = runner.invoke(cogeo, ["validate", raster_path_rgb])
-    assert not result.exception
-    assert result.exit_code == 0
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            cogeo, ["create", raster_path_rgb, "output.tif", "--quiet"]
+        )
+        with rasterio.open("output.tif") as src_dst:
+            print(src_dst.meta)
+
+        result = runner.invoke(cogeo, ["validate", "output.tif"])
+        assert "is a valid cloud optimized GeoTIFF" in result.output
+        assert not result.exception
+        assert result.exit_code == 0
+
+        result = runner.invoke(cogeo, ["validate", raster_invalid_cog])
+        assert "is NOT a valid cloud optimized GeoTIFF" in result.output
+        assert not result.exception
+        assert result.exit_code == 0
diff --git a/tests/test_validate.py b/tests/test_validate.py
index 8ea8e74..7014f5b 100644
--- a/tests/test_validate.py
+++ b/tests/test_validate.py
@@ -31,8 +31,11 @@ jpeg_profile.update({"blockxsize": 256, "blockysize": 256})
 
 def test_cog_validate_valid(monkeypatch):
     """Should work as expected (validate cogeo file)."""
-    # not tiled
-    assert not cog_validate(raster_rgb)
+    # not tiled but 512x512
+    assert cog_validate(raster_rgb)
+
+    # not tiled, no overview
+    assert not cog_validate(raster_big)
 
     # external overview
     assert not cog_validate(raster_external)