Repository: SciTools/cartopy

Issue:
Cartopy cannot be built on a FIPS-enabled system
### Description
<!-- Please provide a general introduction to the issue/proposal. -->
Cython crashes when attempting to build cython extensions to cartopy on FIPS-enabled systems because it tries to use md5 hashes internally, and FIPS-enabled systems disable md5 because it is insecure. This is a Cython issue which I have just filed (Cython fails to build code on FIPS-enabled systems cython/cython#2790), but since cartopy depends on Cython, it became a cartopy issue as well, since it cannot be built on any FIPS-enabled system.

<!--
If you are reporting a bug, attach the *entire* traceback from Python.

If you are proposing an enhancement/new feature, provide links to related articles, reference examples, etc.

If you are asking a question, please ask on StackOverflow and use the cartopy tag. All cartopy
questions on StackOverflow can be found at https://stackoverflow.com/questions/tagged/cartopy
-->

#### Steps to reproduce

1.  RHEL7, FIPS-enabled kernel, python3.4 installed.
2. pip install --prefix $PYTHONUSERBASE cartopy
3. partway through the install, we get the crash and abort seen below:

#### Traceback 

```
    building 'cartopy._crs' extension
    /usr/bin/gcc -Wno-unused-result -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python34/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/opt/rh/rh-python34/root/usr/include -I/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/numpy/core/include -I/opt/rh/rh-python34/root/usr/include/python3.4m -c lib/cartopy/_crs.c -o build/temp.linux-x86_64-3.4/lib/cartopy/_crs.o
    In file included from /opt/rh/rh-python34/root/usr/include/numpy/ndarraytypes.h:1761:0,
                     from /opt/rh/rh-python34/root/usr/include/numpy/ndarrayobject.h:17,
                     from /opt/rh/rh-python34/root/usr/include/numpy/arrayobject.h:4,
                     from lib/cartopy/_crs.c:614:
    /opt/rh/rh-python34/root/usr/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it by " \
      ^
    gcc -pthread -shared -L/opt/rh/rh-python34/root/usr/lib64-Wl,-z,relro build/temp.linux-x86_64-3.4/lib/cartopy/_crs.o -L/opt/rh/rh-python34/root/usr/lib64 -L/opt/rh/rh-python34/root/usr/lib64 -lproj -lpython3.4m -o build/lib.linux-x86_64-3.4/cartopy/_crs.cpython-34m.so
    cythoning lib/cartopy/geodesic/_geodesic.pyx to lib/cartopy/geodesic/_geodesic.c
    /tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bgk3hgjk/cartopy/lib/cartopy/geodesic/_geodesic.pyx
      tree = Parsing.p_module(s, pxd, full_module_name)

    Error compiling Cython file:
    ------------------------------------------------------------
    ...

    cdef extern from *:
        cdef object __pyx_test_dep(object)

    @cname('__pyx_TestClass')
    cdef class TestClass(object):
        ^
    ------------------------------------------------------------

    TestClass:8:5: Compiler crash in AnalyseDeclarationsTransform

    ModuleNode.body = StatListNode(TestClass:4:0)
    StatListNode.stats[1] = CnameDecoratorNode(TestClass:8:5,
        cname = '__pyx_TestClass')

    Compiler crash traceback from this point on:
      File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1635, in visit_CClassDefNode
        self._inject_pickle_methods(node)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1703, in _inject_pickle_methods
        checksum = '0x%s' % hashlib.md5(' '.join(all_members_names).encode('utf-8')).hexdigest()[:7]
    ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
                endpoint(s).

            """

            cdef int n_points, i
            cdef double[:, :] pts, orig_pts
                       ^
    ------------------------------------------------------------

    lib/cartopy/geodesic/_geodesic.pyx:120:20: Compiler crash in AnalyseDeclarationsTransform

    ModuleNode.body = StatListNode(_geodesic.pyx:25:0)
    StatListNode.stats[8] = StatListNode(_geodesic.pyx:46:5)
    StatListNode.stats[0] = CClassDefNode(_geodesic.pyx:46:5,
        as_name = 'Geodesic',
        class_name = 'Geodesic',
        doc = '\n    Define an ellipsoid on which to solve geodesic problems.\n\n    ',
        module_name = '',
        visibility = 'private')
    CClassDefNode.body = StatListNode(_geodesic.pyx:47:4)
    StatListNode.stats[7] = CompilerDirectivesNode(_geodesic.pyx:91:4)
    CompilerDirectivesNode.body = StatListNode(_geodesic.pyx:91:4)
    StatListNode.stats[0] = DefNode(_geodesic.pyx:91:4,
        doc = '\n        Solve the direct geodesic problem where the length of the geodesic is\n        specified in terms of distance.\n\n        Can accept and broadcast length 1 arguments. For example, given a\n        single start point and distance, an array of different azimuths can be\n        supplied to locate multiple endpoints.\n\n        Parameters\n        ----------\n        points: array_like, shape=(n *or* 1, 2)\n            The starting longitude-latitude point(s) from which to travel.\n\n        azimuths: float or array_like with shape=(n, )\n            List of azimuth values (degrees).\n\n        distances : float or array_like with shape(n, )\n            List of distances values (metres).\n\n        Returns\n        -------\n        `numpy.ndarray`, shape=(n, 3)\n            The longitudes, latitudes, and forward azimuths of the located\n            endpoint(s).\n\n        ',
        modifiers = [...]/0,
        name = 'direct',
        needs_closure = True,
        np_args_idx = [...]/0,
        num_required_args = 4,
        outer_attrs = [...]/2,
        py_wrapper_required = True,
        reqd_kw_flags_cname = '0')
    File 'Nodes.py', line 431, in analyse_declarations: StatListNode(_geodesic.pyx:92:8,
        is_terminator = True)
    File 'Nodes.py', line 1329, in analyse_declarations: CVarDefNode(_geodesic.pyx:120:13,
        modifiers = [...]/0,
        visibility = 'private')
    File 'Nodes.py', line 1088, in analyse: MemoryViewSliceTypeNode(_geodesic.pyx:120:20,
        name = 'memoryview')

    Compiler crash traceback from this point on:
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/Nodes.py", line 1084, in analyse
        axes_specs = MemoryView.get_axes_specs(env, self.axes)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/MemoryView.py", line 533, in get_axes_specs
        cythonscope.load_cythonscope()
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/CythonScope.py", line 113, in load_cythonscope
        self, cython_scope=self)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/UtilityCode.py", line 208, in declare_in_scope
        tree = self.get_tree(entries_only=True, cython_scope=cython_scope)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/UtilityCode.py", line 186, in get_tree
        assert not err, err
    AssertionError:
    Error compiling Cython file:
    ------------------------------------------------------------
    ...

    cdef extern from *:
        cdef object __pyx_test_dep(object)

    @cname('__pyx_TestClass')
    cdef class TestClass(object):
        ^
    ------------------------------------------------------------

    TestClass:8:5: Compiler crash in AnalyseDeclarationsTransform

    ModuleNode.body = StatListNode(TestClass:4:0)
    StatListNode.stats[1] = CnameDecoratorNode(TestClass:8:5,
        cname = '__pyx_TestClass')

    Compiler crash traceback from this point on:
      File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1635, in visit_CClassDefNode
        self._inject_pickle_methods(node)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1703, in _inject_pickle_methods
        checksum = '0x%s' % hashlib.md5(' '.join(all_members_names).encode('utf-8')).hexdigest()[:7]
    ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
    building 'cartopy.geodesic._geodesic' extension
    creating build/temp.linux-x86_64-3.4/lib/cartopy/geodesic
    /usr/bin/gcc -Wno-unused-result -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python34/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/opt/rh/rh-python34/root/usr/include -I/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/numpy/core/include -I/opt/rh/rh-python34/root/usr/include/python3.4m -c lib/cartopy/geodesic/_geodesic.c -o build/temp.linux-x86_64-3.4/lib/cartopy/geodesic/_geodesic.o
    lib/cartopy/geodesic/_geodesic.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
     #error Do not use this file, it is the result of a failed Cython compilation.
      ^
    error: command '/usr/bin/gcc' failed with exit status 1

    ----------------------------------------
Command "/opt/rh/rh-python34/root/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-bgk3hgjk/cartopy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-s1x4au0e/install-record.txt --single-version-externally-managed --prefix /usr/appsdev/autoadfc/share/python --compile" failed with error code 1 in /tmp/pip-install-bgk3hgjk/cartopy/

```
<details>
 <summary>Full environment definition</summary>

<!-- fill in the following information as appropriate -->

### Operating system
RedHat EL7

### Cartopy version
Cartopy 0.17.0, the current version on PyPi

### conda list

N/A

### pip list

```
cycler          0.10.0
Cython          0.29.2
docutils        0.12
Jinja2          2.7.3
kiwisolver      1.0.1
MarkupSafe      0.23
matplotlib      2.2.3
nose            1.3.4
numpy           1.15.2
pip             18.0
Pygments        1.6
pyparsing       2.2.2
pyshp           2.0.1
python-dateutil 2.7.3
pytz            2018.5
scipy           0.13.3
setuptools      11.3.1
Shapely         1.6.4.post2
simplejson      3.6.5
six             1.11.0
Sphinx          1.2.2
SQLAlchemy      0.9.8
virtualenv      1.11.6
wheel           0.32.0
```

</details>

Minimum Cython version updated
### Description

It seems that #1251 means that we now depend on Cython 0.29+. 

The compile error I got with 0.28.5 (Linux, GCC 4.4.7):

```
cythoning lib/cartopy/trace.pyx to lib/cartopy/trace.cpp

Error compiling Cython file:
------------------------------------------------------------
...
        else:
            geom = GEOSGeom_createCollection_r(handle, GEOS_MULTILINESTRING,
                                               &geoms[0], geoms.size())
        return geom

    cdef list[Line].size_type size(self):
        ^
------------------------------------------------------------

lib/cartopy/trace.pyx:173:9: 'list[Line].size_type' is not a type identifier
```

We should either try to relax this version dependency, or document it (either is fine IMO).

cc @QuLogic 


Discussion:
As you say, this is an issue with the Cython compiler rather than any code specific to Cartopy cython/cython#2790. What is not quite clear to me is whether this issue only occurs during processing of the pyx file to turn it into C code, or if the C code is trying to use a forbidden function? If it is the former then distributing the C code could be a solution to this, see #1035 for some related discussion.
Including the generated C file(s) in source distributions (and making the recompilation optional) would solve this problem.
That's odd; both [Travis](https://travis-ci.org/SciTools/cartopy/jobs/474287107) and [AppVeyor](https://ci.appveyor.com/project/bjlittle/cartopy/builds/21306953/job/y9pn7pv2se256fk9) build against Cython 0.17.
Hmm, yes indeed. Perhaps it isn't as simple as the Cython version. I was using a pretty old GCC there, so there may be more to it. For others that come across this issue (via Google), upgrading to Cython v0.29 *did* solve the issue for me.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,4 +1,4 @@
-# (C) British Crown Copyright 2011 - 2018, Met Office
+# (C) British Crown Copyright 2011 - 2019, Met Office
 #
 # This file is part of cartopy.
 #
@@ -36,10 +36,20 @@
 import versioneer
 
 
-try:
-    from Cython.Distutils import build_ext
-except ImportError:
-    raise ImportError('Cython 0.17+ is required to install cartopy.')
+# The existence of a PKG-INFO directory is enough to tell us whether this is a
+# source installation or not (sdist).
+HERE = os.path.dirname(__file__)
+IS_SDIST = os.path.exists(os.path.join(HERE, 'PKG-INFO'))
+
+if not IS_SDIST:
+    import Cython
+    if Cython.__version__ < '0.28':
+        raise ImportError(
+            "Cython 0.28+ is required to install cartopy from source.")
+
+    from Cython.Distutils import build_ext as cy_build_ext
+
+
 try:
     import numpy as np
 except ImportError:
@@ -52,8 +62,6 @@
 GEOS_MIN_VERSION = (3, 3, 3)
 PROJ_MIN_VERSION = (4, 9, 0)
 
-HERE = os.path.dirname(__file__)
-
 
 def file_walk_relative(top, remove=''):
     """
@@ -280,8 +288,9 @@ def find_proj_version_by_program(conda=None):
             proj_includes = proj_includes.decode()
             proj_clibs = proj_clibs.decode()
 
-        proj_includes = [proj_include[2:] if proj_include.startswith('-I') else
-                         proj_include for proj_include in proj_includes.split()]      
+        proj_includes = [
+            proj_include[2:] if proj_include.startswith('-I') else
+            proj_include for proj_include in proj_includes.split()]
 
         proj_libraries = []
         proj_library_dirs = []
@@ -321,13 +330,62 @@ def get_config_var(name):
 
 # Description
 # ===========
-
 with open(os.path.join(HERE, 'README.md'), 'r') as fh:
     description = ''.join(fh.readlines())
 
 
+extensions = [
+    Extension(
+        'cartopy.trace',
+        ['lib/cartopy/trace.pyx'],
+        include_dirs=([include_dir, './lib/cartopy', np.get_include()] +
+                      proj_includes + geos_includes),
+        libraries=proj_libraries + geos_libraries,
+        library_dirs=[library_dir] + proj_library_dirs + geos_library_dirs,
+        language='c++',
+        **extra_extension_args),
+    Extension(
+        'cartopy._crs',
+        ['lib/cartopy/_crs.pyx'],
+        include_dirs=[include_dir, np.get_include()] + proj_includes,
+        libraries=proj_libraries,
+        library_dirs=[library_dir] + proj_library_dirs,
+        **extra_extension_args),
+    # Requires proj v4.9
+    Extension(
+        'cartopy.geodesic._geodesic',
+        ['lib/cartopy/geodesic/_geodesic.pyx'],
+        include_dirs=[include_dir, np.get_include()] + proj_includes,
+        libraries=proj_libraries,
+        library_dirs=[library_dir] + proj_library_dirs,
+        **extra_extension_args),
+]
+
+
+def decythonize(extensions, **_ignore):
+    # Remove pyx sources from extensions.
+    # Note: even if there are changes to the pyx files, they will be ignored.
+    for extension in extensions:
+        sources = []
+        for sfile in extension.sources:
+            path, ext = os.path.splitext(sfile)
+            if ext in ('.pyx',):
+                if extension.language == 'c++':
+                    ext = '.cpp'
+                else:
+                    ext = '.c'
+                sfile = path + ext
+            sources.append(sfile)
+        extension.sources[:] = sources
+    return extensions
+
+
 cmdclass = versioneer.get_cmdclass()
-cmdclass.update({'build_ext': build_ext})
+
+if IS_SDIST:
+    extensions = decythonize(extensions)
+else:
+    cmdclass.update({'build_ext': cy_build_ext})
 
 
 # Main setup
@@ -368,36 +426,7 @@ def get_config_var(name):
 
 
     # requires proj headers
-    ext_modules=[
-        Extension(
-            'cartopy.trace',
-            ['lib/cartopy/trace.pyx'],
-            include_dirs=([include_dir, './lib/cartopy', np.get_include()] +
-                          proj_includes + geos_includes),
-            libraries=proj_libraries + geos_libraries,
-            library_dirs=[library_dir] + proj_library_dirs + geos_library_dirs,
-            language='c++',
-            **extra_extension_args
-        ),
-        Extension(
-            'cartopy._crs',
-            ['lib/cartopy/_crs.pyx'],
-            include_dirs=[include_dir, np.get_include()] + proj_includes,
-            libraries=proj_libraries,
-            library_dirs=[library_dir] + proj_library_dirs,
-            **extra_extension_args
-        ),
-        # Requires proj v4.9
-        Extension(
-            'cartopy.geodesic._geodesic',
-            ['lib/cartopy/geodesic/_geodesic.pyx'],
-            include_dirs=[include_dir, np.get_include()] + proj_includes,
-            libraries=proj_libraries,
-            library_dirs=[library_dir] + proj_library_dirs,
-            **extra_extension_args
-        ),
-    ],
-
+    ext_modules=extensions,
     cmdclass=cmdclass,
     python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*',
     classifiers=[


Test patch: