Repository: pyinstaller/pyinstaller

Issue:
Hook for `osgeo` libraries
I am attempting to put together a hook file for `osgeo` libraries (i.e., `gdal`, `ogr`, `osr`, ) that is required to be a bit flexible to adapt at different possible installation flavors.

As first step, I am attempting to manage these two configurations on `Python 2.7.10 32bit`:
- classic `cpython` (using the `gdal` wheel from http://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal with `1.11.3`)
- `miniconda` (thus the `gdal` conda package `1.11.2`)

This latter is a bit more complicated, since the auxiliary data (and the `proj4` library) are stored separately from the python package. Given that, I wrote:

```
from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.utils.misc import is_win

import os
import sys

# osgeo.gdal requires auxiliary data and may have hidden dependencies
# hook was tested with gdal (v.1.11.2 and 1.11.3) on Win7 64bit

is_conda = False

# auxiliary data
# - general case
datas = collect_data_files('osgeo', subdir='data/gdal')

if len(datas) == 0:

    # - conda-specific
    tgt_gdal_data = os.path.join('Library', 'data')
    src_gdal_data = os.path.join(os.path.dirname(sys.executable), 'Library', 'data')
    if os.path.exists(src_gdal_data):
        is_conda = True
        files = os.listdir(src_gdal_data)
        for f in files:
            src_abs_path = os.path.join(src_gdal_data, f)
            datas.append((src_abs_path, tgt_gdal_data))

if is_conda:
    if is_win:
        proj_dll = os.path.join(os.path.dirname(sys.executable), 'proj.dll')
        binaries = [(proj_dll, ""), ]
```

Such a hook seems to work for these 2 specific configurations. However, I am not sure that I am following the "best practices" on doing this.

Which of the existing hook files do you suggest to follow as template? 
Do you have directions on how to better manage the `osgeo` hook given the several possible configurations? 



Discussion:
Some notes:
- use import `from PyInstaller.compat import is_win`
- add more comments about what is conda and where it stores data files
- `sys.executable` won't work in virtualenv. To obtain data dir, rather use `sys.real_prefix`.
- put there  a `TODO` that it should be tested on other platforms.
- You could replace this whole code 

``` python
        files = os.listdir(src_gdal_data)
        for f in files:
            src_abs_path = os.path.join(src_gdal_data, f)
            datas.append((src_abs_path, tgt_gdal_data))
```

with just

``` python
datas.append((src_gdal_data, tgt_gdal_data))
```

Using `gdal 2.0.0`, it freezes but I get an error when I run the application. This is the full trace:

```
PyInstaller Bootloader 3.x
LOADER: executable is C:\zzzz\test_data\gdal\dist\minimal\minimal.exe
LOADER: homepath is C:\zzzz\test_data\gdal\dist\minimal
LOADER: _MEIPASS2 is NULL
LOADER: archivename is C:\zzzz\test_data\gdal\dist\minimal\minimal.exe
LOADER: No need to extract files to run; setting extractionpath to homepath
LOADER: SetDllDirectory(C:\zzzz\test_data\gdal\dist\minimal)
LOADER: Already in the child - running user's code.
LOADER: Python library: C:\zzzz\test_data\gdal\dist\minimal\python27.dll
LOADER: Loaded functions from Python library.
LOADER: Manipulating environment (sys.path, sys.prefix)
LOADER: sys.prefix is C:\zzzz\test_data\gdal\dist\minimal
LOADER: Setting runtime options
LOADER: Initializing python
LOADER: Overriding Python's sys.path
LOADER: Post-init sys.path is C:\zzzz\test_data\gdal\dist\minimal
LOADER: Setting sys.argv
LOADER: setting sys._MEIPASS
LOADER: importing modules from CArchive
LOADER: extracted struct
LOADER: callfunction returned...
LOADER: extracted pyimod01_os_path
LOADER: callfunction returned...
LOADER: extracted pyimod02_archive
LOADER: callfunction returned...
LOADER: extracted pyimod03_importers
LOADER: callfunction returned...
LOADER: Installing PYZ archive with Python modules.
LOADER: PYZ archive: out00-PYZ.pyz
LOADER: Running pyiboot01_bootstrap.py
LOADER: Running pyiboot02_egg_install.py
LOADER: Running pyi_rth__tkinter.py
LOADER: Running pyi_rth_win32comgenpy.py
LOADER: Running pyi_rth_pkgres.py
LOADER: Running minimal.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\users\xxxx\appdata\local\yyyy\py2\lib\site-packages\PyInstaller-3.0.dev6+b7954b3-py2.7.egg\PyInstaller\loader\pyimod03_importers.py", line 313, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\xxxx\AppData\Local\yyyy\py2\lib\site-packages\osr.py", line 2, in <module>
    from osgeo.gdal import deprecation_warn
  File "c:\users\xxxx\appdata\local\yyyy\py2\lib\site-packages\PyInstaller-3.0.dev6+b7954b3-py2.7.egg\PyInstaller\loader\pyimod03_importers.py", line 313, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\xxxx\AppData\Local\yyyy\py2\lib\site-packages\osgeo\gdal.py", line 427, in <module>
    import ogr
  File "c:\users\xxxx\appdata\local\yyyy\py2\lib\site-packages\PyInstaller-3.0.dev6+b7954b3-py2.7.egg\PyInstaller\loader\pyimod03_importers.py", line 313, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\xxxx\AppData\Local\yyyy\py2\lib\site-packages\osgeo\ogr.py", line 251, in <module>
    _MyHelper(_Module())
  File "C:\Users\xxxx\AppData\Local\yyyy\py2\lib\site-packages\osgeo\ogr.py", line 219, in __init__
    self.original_help = help
NameError: global name 'help' is not defined
minimal returned -1
LOADER: OK.
LOADER: Cleaning up Python interpreter.
```

The code snippet causing the exception is the SWIG-generated `ogr.py`:

```
# Custom help() replacement to display the help of the original module
# instead of the one of our instance object
class _MyHelper(object):

    def __init__(self, module):
        self.module = module
        self.original_help = help

        # Replace builtin help by ours
        try:
            import __builtin__ as builtins # Python 2
        except ImportError:
            import builtins # Python 3
        builtins.help = self
```

 It looks like it does not find the built-in `help` function.. In `gdal 1.11.2` I didn't find nothing similar to this class.

So far, this is where I am with the `gdal` hook for vanilla and Anaconda `cpython` (tested also on Yosemite and Ubuntu):

```
from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.compat import is_win, is_darwin

import os
import sys

# The osgeo libraries require auxiliary data and may have hidden dependencies.
# There are several possible configurations on how these libraries can be 
# deployed.
# This hook evaluates the cases when:
# - the `data` folder is present "in-source" (sharing the same namespace folder
#   as the code libraries)
# - the `data` folder is present "out-source" (for instance, on Anaconda for 
#   Windows, in PYTHONHOME/Library/data)
# In this latter case, the hook also checks for the presence of `proj` library 
# (e.g., on Windows in PYTHONHOME) for being added to the bundle.
#
# This hook has been tested with gdal (v.1.11.2 and 1.11.3) on:
# - Win7 64bit
# - Ubuntu 15.04 64bit
# - Mac OS X Yosemite 10.10
#
# TODO: Fix for gdal>=2.0: 'NameError: global name 'help' is not defined'

# flag used to identify an Anaconda environment
is_conda = False

# Auxiliary data:
#
# - general case (data in 'osgeo/data/gdal'):
datas = collect_data_files('osgeo', subdir='data/gdal')

# check if the data has been effectively found in 'osgeo/data/gdal'
if len(datas) == 0:

    if hasattr(sys, 'real_prefix'):  # check if in a virtual environment
        root_path = sys.real_prefix
    else:
        root_path = sys.prefix

    # - conda-specific
    if is_win:
        tgt_gdal_data = os.path.join('Library', 'data')
        src_gdal_data = os.path.join(root_path, 'Library', 'data')
    else:  # both linux and darwin
        tgt_gdal_data = os.path.join('share', 'gdal')
        src_gdal_data = os.path.join(root_path, 'share', 'gdal')
        # TODO: need to add to the bundle something like:
        # "import os; os.environ['GDAL_DATA']='%s/share/gdal' % sys._MEIPASS"

    if os.path.exists(src_gdal_data):
        is_conda = True
        datas.append((src_gdal_data, tgt_gdal_data))

# Hidden dependencies
if is_conda:
    # if `proj.4` is present, it provides additional functionalities
    if is_win:
        proj4_lib = os.path.join(root_path, 'proj.dll')
    elif is_darwin:
        proj4_lib = os.path.join(root_path, 'lib', 'libproj.dylib')
    else:  # assumed linux-like settings
        proj4_lib = os.path.join(root_path, 'lib', 'libproj.so')

    if os.path.exists(proj4_lib):
        binaries = [(proj4_lib, ""), ]
```

@matysek: I have added another `TODO`. This is because I don't know how to add the creation of an os environ from the hook (`GDAL_DATA`). Is this currently possible without modifying the user code?

> I have added another TODO. This is because I don't know how to add the creation of an os environ from the hook (GDAL_DATA). Is this currently possible without modifying the user code?

Yes, for this purpose there are runtime hooks. See:
- PyInstaller/loader/rthooks
- PyInstaller/loader/rthooks.dat

For example the matplotlib rthook does it too.

> Using gdal 2.0.0, it freezes but I get an error when I run the application. 

@matysek , do you prefer to keep open this ticket for this issue related to `osgeo >=2.0` or to close it for opening an issue-specific ticket? 

@giumas let's this ticket open eve for >=2.0.

I have just attempted to run a minimal file (`from osgeo import ogr`) with the current stable release (`3.1`), but I got the same issue:

```
PyInstaller Bootloader 3.x
LOADER: executable is C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal\minimal.exe
LOADER: homepath is C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal
LOADER: _MEIPASS2 is NULL
LOADER: archivename is C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal\minimal.exe
LOADER: No need to extract files to run; setting extractionpath to homepath
LOADER: SetDllDirectory(C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal)
LOADER: Already in the child - running user's code.
LOADER: Python library: C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal\python34.dll
LOADER: Loaded functions from Python library.
LOADER: Manipulating environment (sys.path, sys.prefix)
LOADER: Pre-init sys.path is C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal\base_library.zip;C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal
LOADER: sys.prefix is C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal
LOADER: Setting runtime options
LOADER: Initializing python
LOADER: Overriding Python's sys.path
LOADER: Post-init sys.path is C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal\base_library.zip;C:\_yyyyyy\zzz_test\gdal_freeze\dist\minimal
LOADER: Setting sys.argv
LOADER: setting sys._MEIPASS
LOADER: importing modules from CArchive
LOADER: extracted struct
LOADER: callfunction returned...
LOADER: extracted pyimod01_os_path
LOADER: callfunction returned...
LOADER: extracted pyimod02_archive
LOADER: callfunction returned...
LOADER: extracted pyimod03_importers
LOADER: callfunction returned...
LOADER: Installing PYZ archive with Python modules.
LOADER: PYZ archive: out00-PYZ.pyz
LOADER: Running pyiboot01_bootstrap.py
LOADER: Running pyi_rth_osgeo.py
LOADER: Running pyi_rth_pkgres.py
LOADER: Running pyi_rth_win32comgenpy.py
LOADER: Running minimal.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "c:\users\xxxxx\appdata\local\hyo\py3\x64\envs\gdal20\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
    exec(bytecode, module.__dict__)
  File "site-packages\ogr.py", line 2, in <module>
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "c:\users\xxxxx\appdata\local\hyo\py3\x64\envs\gdal20\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
    exec(bytecode, module.__dict__)
  File "site-packages\osgeo\gdal.py", line 427, in <module>
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "c:\users\xxxxx\appdata\local\hyo\py3\x64\envs\gdal20\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
    exec(bytecode, module.__dict__)
  File "site-packages\osgeo\ogr.py", line 251, in <module>
  File "site-packages\osgeo\ogr.py", line 219, in __init__
NameError: name 'help' is not defined
minimal returned -1
LOADER: OK.
LOADER: Cleaning up Python interpreter.
```

Googling around, I have found this post from @rouault: http://erouault.blogspot.com/2014/11/hacking-python-module-constants.html that explains what was the reason to introduce the trick that is giving troubles to `PyInstaller`.

Unluckily, the problem is not going to go away with more recent `gdal` release since this part of `ogr.py` is currently the same: https://github.com/OSGeo/gdal/blob/trunk/gdal/swig/python/osgeo/ogr.py#L222

Do you have any idea on why the frozen application fails stating that the builtin `help` is not defined?

I have got the same issue:  "global name 'help' is not defined"
I use pyinstaller 3.1.1

I've got the same issue working on my [DicoGIS](https://github.com/Guts/DicoGIS) project.
I reported it in the OSGeo tracker: https://trac.osgeo.org/gdal/ticket/6364

This should be finally fixed with the release of GDAL 2.0.3: https://trac.osgeo.org/gdal/ticket/6364#comment:4


Resolving patch:
diff --git a/PyInstaller/lib/modulegraph/modulegraph.py b/PyInstaller/lib/modulegraph/modulegraph.py
--- a/PyInstaller/lib/modulegraph/modulegraph.py
+++ b/PyInstaller/lib/modulegraph/modulegraph.py
@@ -2219,6 +2219,16 @@ def _safe_import_hook(
         """
         self.msg(3, "_safe_import_hook", target_module_partname, source_module, target_attr_names, level)
 
+        def is_swig_candidate():
+            return (source_module is not None and
+                    target_attr_names is None and
+                    level == ABSOLUTE_IMPORT_LEVEL and
+                    type(source_module) is SourceModule and
+                    target_module_partname ==
+                      '_' + source_module.identifier.rpartition('.')[2] and
+                    sys.version_info[0] == 3)
+            
+
         # List of the graph nodes created for all target modules both
         # imported by and returned from this call, whose:
         #
@@ -2272,14 +2282,7 @@ def _safe_import_hook(
             #
             # Only source modules (e.g., ".py"-suffixed files) are SWIG import
             # candidates. All other node types are safely ignorable.
-            if (
-                source_module is not None and
-                target_attr_names is None and
-                level == ABSOLUTE_IMPORT_LEVEL and
-                type(source_module) is SourceModule and
-                target_module_partname ==
-                    '_' + source_module.identifier.rpartition('.')[2] and
-                sys.version_info[0] == 3):
+            if is_swig_candidate():
                 self.msg(4, 'SWIG import candidate (name=%r, caller=%r, level=%r)' % (target_module_partname, source_module, level))
 
                 # TODO Define a new function util.open_text_file() performing
@@ -2339,6 +2342,25 @@ def _safe_import_hook(
         # Target module imported above.
         target_module = target_modules[0]
 
+        if isinstance(target_module, MissingModule) and is_swig_candidate():
+            # if this possible swig C module was previously imported from
+            # a python module other than its corresponding swig python
+            # module, then it may have been considered a MissingModule.
+            # Try to reimport it now. For details see pull-request #2578
+            # and issue #1522.
+            # Remove the MissingModule node from the graph so that we can
+            # attempt a reimport and avoid collisions. This node should be
+            # fine to remove because the proper module will be imported and
+            # added to the graph in the next line (call to _safe_import_hook).
+            self.removeNode(target_module)
+            # Reimport the SWIG C module
+            target_modules = self._safe_import_hook(
+                target_module_partname, source_module,
+                target_attr_names=None, level=level, edge_attr=edge_attr)
+            # return the output regardless because it would just be
+            # duplicating the processing below
+            return target_modules
+
         if isinstance(edge_attr, DependencyInfo):
             edge_attr = edge_attr._replace(fromlist=True)
 


Test patch: