Repository: spack/spack

Issue:
gdal@3.3.2+zstd not working as expected
When installing `gdal` with `zstd` compression support, build and installation look fine, but `gdal_translate -co COMPRESS=ZSTD` gives
```
Warning 6: 'ZSTD' is an unexpected value for COMPRESS creation option of type string-select.
ERROR 1: Cannot create TIFF file due to missing codec for ZSTD.
```
I attach my `build-env` and `build-out`
[spack-build-env.txt](https://github.com/spack/spack/files/7366926/spack-build-env.txt)
[spack-build-out.txt](https://github.com/spack/spack/files/7366927/spack-build-out.txt)
The shown config messages look fine:
```
checking for ZSTD_decompressStream in -lzstd... yes
...
ZSTD support:              yes
```
the dependency is installed, but then the library doesn't seem to be linked correctly (at least this is my first guess).

``` console
spack --version
0.16.3-4766-449a583
```
and I got the same issue testing on 2 different CentOS systems.

Any help from @adamjstewart or others would be greatly appreciated! Thanks!


Discussion:
Thanks for the bug report, I was able to reproduce this on macOS too. Based on the [GeoTIFF driver docs](https://gdal.org/drivers/raster/gtiff.html#creation-options):

> ZSTD is available since GDAL 2.3 when using internal libtiff and if GDAL built against libzstd >=1.0, or if built against external libtiff with zstd support.

It looks like the problem is that we are using an external libtiff that isn't compiled with zstd support. I added a zstd variant to the libtiff package. Testing this now and will submit a PR to fix this when I'm done.
That sounds very promising, thanks for looking into this!

I should probably open a separate "issue" about this, but is it possible that something similar is happening when trying to add a `--with-lerc=...` compiler option (to get `lerc` compression support, see [https://github.com/OSGeo/gdal/blob/master/gdal/configure.ac](https://github.com/OSGeo/gdal/blob/master/gdal/configure.ac), [https://github.com/Esri/lerc](https://github.com/Esri/lerc) )? Just in case you are familiar with compiling `gdal` with this option.
Correct:

> LERC and LERC_DEFLATE are available only when using internal libtiff for GDAL < 3.3.0. Since GDAL 3.3.0, LERC compression is also available when building GDAL against external libtiff >= 4.3.0, built itself against https://github.com/esri/lerc

I can look into adding a LERC package to Spack.
If possible, that would be amazing, thank you! 
In the meantime, I should compile `libtiff` with both `zstd` and `lerc` support (with the latter being compiled separately, while unavailable in spack)... interesting, thanks!
Correct. This shouldn't take me more than an hour.
I can definitely wait, then! Thanks a lot for your help!

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/gdal/package.py b/var/spack/repos/builtin/packages/gdal/package.py
--- a/var/spack/repos/builtin/packages/gdal/package.py
+++ b/var/spack/repos/builtin/packages/gdal/package.py
@@ -58,7 +58,6 @@ class Gdal(AutotoolsPackage):
     variant('libz',      default=True,  description='Include libz support')
     variant('libiconv',  default=False, description='Include libiconv support')
     variant('liblzma',   default=True,  description='Include liblzma support')
-    variant('zstd',      default=False, description='Include zstd support')
     variant('pg',        default=False, description='Include PostgreSQL support')
     variant('cfitsio',   default=False, description='Include FITS support')
     variant('png',       default=False, description='Include PNG support')
@@ -117,7 +116,6 @@ class Gdal(AutotoolsPackage):
     depends_on('zlib', when='+libz')
     depends_on('iconv', when='+libiconv')
     depends_on('xz', when='+liblzma')
-    depends_on('zstd', when='+zstd @2.3:')
     depends_on('postgresql', when='+pg')
     depends_on('cfitsio', when='+cfitsio')
     depends_on('libpng', when='+png')
@@ -265,11 +263,6 @@ def configure_args(self):
                 args.append('--with-mrf=no')
 
         if spec.satisfies('@2.3:'):
-            if '+zstd' in spec:
-                args.append('--with-zstd={0}'.format(spec['zstd'].prefix))
-            else:
-                args.append('--with-zstd=no')
-
             if '+proj' in spec:
                 args.append('--with-proj={0}'.format(spec['proj'].prefix))
             else:
diff --git a/var/spack/repos/builtin/packages/lerc/package.py b/var/spack/repos/builtin/packages/lerc/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin/packages/lerc/package.py
@@ -0,0 +1,22 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack import *
+
+
+class Lerc(CMakePackage):
+    """Limited Error Raster Compression.
+
+    LERC is an open-source image or raster format which supports rapid encoding
+    and decoding for any pixel type (not just RGB or Byte). Users set the maximum
+    compression error per pixel while encoding, so the precision of the original
+    input image is preserved (within user defined error bounds)."""
+
+    homepage = "https://github.com/Esri/lerc"
+    url      = "https://github.com/Esri/lerc/archive/refs/tags/v3.0.tar.gz"
+
+    version('3.0', sha256='8c0148f5c22d823eff7b2c999b0781f8095e49a7d3195f13c68c5541dd5740a1')
+
+    depends_on('cmake@3.11:', type='build')
diff --git a/var/spack/repos/builtin/packages/libtiff/package.py b/var/spack/repos/builtin/packages/libtiff/package.py
--- a/var/spack/repos/builtin/packages/libtiff/package.py
+++ b/var/spack/repos/builtin/packages/libtiff/package.py
@@ -12,6 +12,9 @@ class Libtiff(AutotoolsPackage):
     homepage = "http://www.simplesystems.org/libtiff/"
     url      = "https://download.osgeo.org/libtiff/tiff-4.1.0.tar.gz"
 
+    maintainers = ['adamjstewart']
+
+    version('4.3.0',  sha256='0e46e5acb087ce7d1ac53cf4f56a09b221537fc86dfc5daaad1c2e89e1b37ac8')
     version('4.2.0',  sha256='eb0484e568ead8fa23b513e9b0041df7e327f4ee2d22db5a533929dfc19633cb')
     version('4.1.0',  sha256='5d29f32517dadb6dbcd1255ea5bbc93a2b54b94fbf83653b4d65c7d6775b8634')
     version('4.0.10', sha256='2c52d11ccaf767457db0c46795d9c7d1a8d8f76f68b0b800a3dfe45786b996e4')
@@ -21,12 +24,59 @@ class Libtiff(AutotoolsPackage):
     version('4.0.6',  sha256='4d57a50907b510e3049a4bba0d7888930fdfc16ce49f1bf693e5b6247370d68c')
     version('3.9.7',  sha256='f5d64dd4ce61c55f5e9f6dc3920fbe5a41e02c2e607da7117a35eb5c320cef6a')
 
-    depends_on('jpeg')
-    depends_on('zlib')
-    depends_on('xz')
+    variant('zlib', default=False, description='Enable Zlib usage')
+    variant('libdeflate', default=False, description='Enable libdeflate usage')
+    variant('pixarlog', default=False, description='Enable support for Pixar log-format algorithm')
+    variant('jpeg', default=False, description='Enable IJG JPEG library usage')
+    variant('old-jpeg', default=False, description='Enable support for Old JPEG compression')
+    variant('jpeg12', default=False, description='Enable libjpeg 8/12bit dual mode')
+    variant('jbig', default=False, description='Enable JBIG-KIT usage')
+    variant('lerc', default=False, description='Enable liblerc usage')
+    variant('lzma', default=False, description='Enable liblzma usage')
+    variant('zstd', default=False, description='Enable libzstd usage')
+    variant('webp', default=False, description='Enable libwebp usage')
+
+    depends_on('zlib', when='+zlib')
+    depends_on('zlib', when='+pixarlog')
+    depends_on('jpeg@5:', when='+jpeg')
+    depends_on('jbigkit', when='+jbig')
+    depends_on('lerc', when='+lerc')
+    depends_on('xz', when='+lzma')
+    depends_on('zstd@1:', when='+zstd')
+    depends_on('libwebp', when='+webp')
+
+    conflicts('+libdeflate', when='~zlib')
+    conflicts('+libdeflate', when='@:4.1')
+    conflicts('+jpeg12', when='~jpeg')
+    conflicts('+jpeg12', when='@:3')
+    conflicts('+lerc', when='~zlib')
+    conflicts('+lerc', when='@:4.2')
+    conflicts('+lzma', when='@:3')
+    conflicts('+zstd', when='@:4.0.9')
+    conflicts('+webp', when='@:4.0.9')
 
     def patch(self):
         # Remove flags not recognized by the NVIDIA compiler
         if self.spec.satisfies('%nvhpc@:20.11'):
             filter_file('vl_cv_prog_cc_warnings="-Wall -W"',
                         'vl_cv_prog_cc_warnings="-Wall"', 'configure')
+
+    def configure_args(self):
+        args = []
+        args += self.enable_or_disable('zlib')
+        if self.spec.satisfies('@4.2:'):
+            args += self.enable_or_disable('libdeflate')
+        args += self.enable_or_disable('pixarlog')
+        args += self.enable_or_disable('jpeg')
+        args += self.enable_or_disable('old-jpeg')
+        if self.spec.satisfies('@4:'):
+            args += self.enable_or_disable('jpeg12')
+        args += self.enable_or_disable('jbig')
+        if self.spec.satisfies('@4.3:'):
+            args += self.enable_or_disable('lerc')
+        if self.spec.satisfies('@4:'):
+            args += self.enable_or_disable('lzma')
+        if self.spec.satisfies('@4.0.10:'):
+            args += self.enable_or_disable('zstd')
+            args += self.enable_or_disable('webp')
+        return args


Test patch: