Repository: pyinstaller/pyinstaller

Issue:
SciPy >= 0.15.0 Hidden Import: `scipy.integrate`
In SciPy >= 0.15.0:
1. The `scipy.special.__init__` module imports...
2. The `scipy.special._ellip_harm` module imports...
3. The `scipy.special._ellip_harm_2` C extension imports...
4. The `scipy.integrate` package.

The third import is undetectable by PyInstaller. Attempting to import `scipy.special` within frozen applications raises runtime exceptions resembling:

```
Traceback (most recent call last):
  File "site-packages/scipy/interpolate/__init__.py", line 145, in <module>
  File "/home/leycec/py/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
    exec(bytecode, module.__dict__)
  File "site-packages/scipy/interpolate/interpolate.py", line 16, in <module>
  File "/home/leycec/py/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
    exec(bytecode, module.__dict__)
  File "site-packages/scipy/special/__init__.py", line 608, in <module>
  File "/home/leycec/py/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
    exec(bytecode, module.__dict__)
  File "site-packages/scipy/special/_ellip_harm.py", line 7, in <module>
  File "/home/leycec/py/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 573, in load_module
    module = loader.load_module(fullname)
  File "scipy/special/_ellip_harm_2.pyx", line 6, in init scipy.special._ellip_harm_2 (scipy/special/_ellip_harm_2.c:7441)
ImportError: No module named 'scipy.integrate'
```

Existing issues #1679, #1680, and #1742 all show users passing `--hidden-import=scipy.integrate` to `pyinstaller`, implying this to be a fairly widespread (albeit trivial) issue. I'll submit a pull request putting this problem to the sword... _any femtosecond now._

**Merry Easter Bunny Day.**



Discussion:

Resolving patch:
diff --git a/PyInstaller/hooks/hook-scipy.special._ellip_harm_2.py b/PyInstaller/hooks/hook-scipy.special._ellip_harm_2.py
new file mode 100644
--- /dev/null
+++ b/PyInstaller/hooks/hook-scipy.special._ellip_harm_2.py
@@ -0,0 +1,31 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2013-2016, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License with exception
+# for distributing bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#-----------------------------------------------------------------------------
+
+"""
+Module hook for the `scipy.special._ellip_harm_2` C extension first introduced
+by SciPy >= 0.15.0.
+
+See Also
+----------
+https://github.com/scipy/scipy/blob/master/scipy/special/_ellip_harm_2.pyx
+    This C extension's Cython-based implementation.
+"""
+
+# In SciPy >= 0.15.0:
+#
+# 1. The "scipy.special.__init__" module imports...
+# 2. The "scipy.special._ellip_harm" module imports...
+# 3. The "scipy.special._ellip_harm_2" C extension imports...
+# 4. The "scipy.integrate" package.
+#
+# The third import is undetectable by PyInstaller and hence explicitly listed.
+# Since "_ellip_harm" and "_ellip_harm_2" were first introduced by SciPy 0.15.0,
+# the following hidden import will only be applied for versions of SciPy
+# guaranteed to provide these modules and C extensions.
+hiddenimports = ['scipy.integrate']


Test patch:
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,108 @@
+Tests for PyInstaller
+=====================
+
+This directory contains tests for PyInstaller:
+
+- `functional` directory contains tests where executables are created from
+  Python scripts.
+- `old_suite` directory contains old structure of tests (TODO migrate all tests
+  to a new structure).
+- `unit` directory contains simple unit tests.
+
+Prerequisites
+-------------
+
+In order to run the tests, you will need the following Python packages/libraries
+installed:
+
+- Mock
+- pytest
+
+Running the Tests
+-----------------
+
+To run the tests, navigate to the root directory of the PyInstaller project and
+run the following command:
+
+    py.test
+
+Or, to speed up test runs by sending tests to multiple CPUs:
+
+    py.test -n NUM
+
+Or, to run only the unit or functional tests, run the following command:
+
+    TODO
+
+Or, to run only a particular test suite within a file, run the following
+command:
+
+    TODO
+
+Run all tests matching `test_ctypes_CDLL` resp. `ctypes_CDLL`:
+
+    py.test -k test_ctypes_CDLL
+    py.test -k ctypes_CDLL
+
+Run both the onefile and ondir tests for
+`test_ctypes_CDLL_find_library__nss_files`:
+
+    py.test -k test_ctypes_CDLL_find_library__nss_files
+
+Finally, to only run a particular test, run one of the following commands:
+
+    py.test -k test_ctypes_CDLL_find_library__nss_files[onedir]
+    py.test -k test_ctypes_CDLL_find_library__nss_files[onefile]
+
+## Continuous Integration (CI)
+
+Continuous integration (CI) automatically exercises all tests for all platforms
+officially supported by PyInstaller.
+
+### Python Packages
+
+Regardless of platform or CI service, cross-platform Python packages to be
+tested should be listed in `test/requirements-library.txt` using the usual
+`requirements.txt` syntax (e.g., `{package_name}>={minimum_version}`). These
+packages will be installed with `pip` into remote testing environments managed
+by third-party CI services for _all_ supported platforms.
+
+Platform-specific Python packages to be tested should instead be:
+
+- For OS X, listed in `test/requirements-mac.txt`.
+- For Windows, listed in `test/requirements-win.txt`.
+
+Cross-platform Python packages required for exercising tests (e.g., `pytest`)
+should instead be listed in `test/requirements-tools.txt`.
+
+### Linux
+
+The top-level `.travis.yml` file configures the Travis-CI service to remotely
+test PyInstaller in an Ubuntu 12.04 (LTS) container, the most recent Linux
+distribution supported by Travis-CI.
+
+Non-Python dependencies installable through `apt-get` on Ubuntu 12.04 should be
+listed as `- `-prefixed items in the `addons:` → `apt:` → `packages:` subsection
+of `.travis.yml`. Since Ubuntu 12.04 provides _no_ Python 3 packages prefixed by
+`python3-`, only Python 2.7 packages prefixed by `python-` are installable by
+`apt-get`. Since installing only Python 2.7 packages would be useless, Python
+packages should _always_ be installed by `pip` rather than `apt-get`. See
+**"Python Packages"** above.
+
+### OS X
+
+The top-level `.travis.yml` file of a
+[separate repository](https://github.com/pyinstaller/pyinstaller-osx-tests)
+configures the Travis-CI service to remotely test PyInstaller in an OS X 10.9.5
+virtual machine, the most recent OS X version supported by Travis-CI.
+
+### Windows
+
+The top-level `appveyor.yml` file configures the Appveyor service to remotely
+test PyInstaller in a Windows virtual machine.
+
+Non-Python dependencies installable through either Chocolatey (`cinst`),
+PowerShell (`ps`), or WebPI (`WebpiCmd`) should be listed as `- `-prefixed items
+in the `install:` section of `appveyor.yml`. See the
+[official documentation](http://www.appveyor.com/docs/build-configuration#installing-additional-software)
+for voluminous details.
diff --git a/tests/README.txt b/tests/README.txt
deleted file mode 100644
--- a/tests/README.txt
+++ /dev/null
@@ -1,52 +0,0 @@
-Tests for PyInstaller
-=====================
-
-This directory contains tests for PyInstaller:
-
- - ``functional`` directory contains tests where executables are created from Python scripts.
- - ``old_suite`` directory contains old structure of tests (TODO migrate all tests to a new structure)
- - ``unit`` directory contains simple unit tests
-
-Prerequisites
--------------
-
-In order to run the tests, you will need the following Python packages/libraries installed:
-
- - Mock
- - pytest
-
-Running the Tests
------------------
-
-To run the tests, navigate to the root directory of the PyInstaller project, and then run the following command::
-
-    py.test
-
-Or, to speed up test runs by sending tests to multiple CPUs
-
-    py.test -n NUM
-
-Or, to run only the unit or functional tests, run the following command::
-
-    TODO
-
-Or, to run only a particular test suite within a file, run the
-following command::
-
-    TODO
-
-Run all tests matching `test_ctypes_CDLL` resp. `ctypes_CDLL`::
-
-    py.test -k test_ctypes_CDLL
-    py.test -k ctypes_CDLL
-
-Run both the onefile and ondir tests for
-`test_ctypes_CDLL_find_library__nss_files`::
-
-    py.test -k test_ctypes_CDLL_find_library__nss_files
-
-Finally, to only run a particular test, run one of the following
-commands::
-
-    py.test -k test_ctypes_CDLL_find_library__nss_files[onedir]
-    py.test -k test_ctypes_CDLL_find_library__nss_files[onefile]
diff --git a/tests/functional/test_hooks/test_scipy.py b/tests/functional/test_hooks/test_scipy.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/test_hooks/test_scipy.py
@@ -0,0 +1,61 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2005-2016, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License with exception
+# for distributing bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#-----------------------------------------------------------------------------
+
+"""
+Functional tests for SciPy.
+"""
+
+from PyInstaller.compat import is_darwin, is_win
+from PyInstaller.utils.tests import importorskip, skip, xfail
+
+
+@xfail(is_win, reason='Issue scipy/scipy#5461.')
+@xfail(is_darwin, reason='Issue #1895.')
+@importorskip('scipy')
+def test_scipy(pyi_builder):
+    pyi_builder.test_source(
+        """
+        from distutils.version import LooseVersion
+
+        # Test top-level SciPy importability.
+        import scipy
+        from scipy import *
+
+        # Test hooked SciPy modules.
+        import scipy.io.matlab
+        import scipy.sparse.csgraph
+
+        # Test problematic SciPy modules.
+        import scipy.linalg
+        import scipy.signal
+
+        # SciPy >= 0.16 privatized the previously public "scipy.lib" package as
+        # "scipy._lib". Since this package is problematic, test its
+        # importability regardless of SciPy version.
+        if LooseVersion(scipy.__version__) >= LooseVersion('0.16.0'):
+            import scipy._lib
+        else:
+            import scipy.lib
+        """)
+
+
+@skip(reason='Issue #1919.')
+@xfail(is_win, reason='Issue scipy/scipy#5461.')
+@xfail(is_darwin, reason='Issue #1895.')
+@importorskip('scipy')
+def test_scipy_special(pyi_builder):
+    """
+    Test the importability of the `scipy.special` package and related hooks.
+
+    This importation _must_ be tested independent of the importation of all
+    other problematic SciPy packages and modules. Combining this test with other
+    SciPy tests (e.g., `test_scipy()`) fails to properly exercise the hidden
+    imports required by this package.
+    """
+    pyi_builder.test_source("""import scipy.special""")
diff --git a/tests/functional/test_libraries.py b/tests/functional/test_libraries.py
--- a/tests/functional/test_libraries.py
+++ b/tests/functional/test_libraries.py
@@ -491,35 +491,6 @@ def testit():
         """)
 
 
-@xfail(is_darwin, reason='Issue #1895.')
-@importorskip('scipy')
-def test_scipy(pyi_builder):
-    pyi_builder.test_source(
-        """
-        from distutils.version import LooseVersion
-
-        # Test top-level SciPy importability.
-        import scipy
-        from scipy import *
-
-        # Test hooked SciPy modules.
-        import scipy.io.matlab
-        import scipy.sparse.csgraph
-
-        # Test problematic SciPy modules.
-        import scipy.linalg
-        import scipy.signal
-
-        # SciPy >= 0.16 privatized the previously public "scipy.lib" package as
-        # "scipy._lib". Since this package is problematic, test its
-        # importability regardless of SciPy version.
-        if LooseVersion(scipy.__version__) >= LooseVersion('0.16.0'):
-            import scipy._lib
-        else:
-            import scipy.lib
-        """)
-
-
 @importorskip('sqlalchemy')
 def test_sqlalchemy(pyi_builder):
     pyi_builder.test_source(
diff --git a/tests/requirements-libraries.txt b/tests/requirements-libraries.txt
--- a/tests/requirements-libraries.txt
+++ b/tests/requirements-libraries.txt
@@ -1,7 +1,13 @@
-# Test libraries
+# Platform-agnostic test libraries.
+#
+# Libraries listed below are unconditionally installed by all continuous
+# integration (CI) services, regardless of platform. Each platform-specific
+# requirements file (e.g., "requirements-linux.txt") includes this file.
+
 # Url for downloading PyCrypto prebuilt Windows binaries:
 # http://www.voidspace.org.uk/python/pycrypto-2.6.1/
 pycrypto  # Required for crypto feature - encrypting bytecode.
+
 Django<1.8.11 # PyInstaller doesn't yet work with Django 1.9
 babel>2.0  # Required by sphinx. Version 2.0 doesn't work on Windows.
 boto
diff --git a/tests/requirements-linux.txt b/tests/requirements-linux.txt
new file mode 100644
--- /dev/null
+++ b/tests/requirements-linux.txt
@@ -0,0 +1,12 @@
+# Linux-specific test libraries.
+#
+# Libraries listed below are only installed by Linux-specific continuous
+# integration (CI) services (e.g., Travis). Platform-agnostic libraries are
+# listed in the "requirements-libraries.txt" file included below.
+
+# Install all platform-agnostic libraries.
+-r requirements-libraries.txt
+
+# SciPy currently provides no binary wheel for Windows and hence must be
+# isolated to non-Windows platforms. See also: scipy/scipy#5461
+scipy
diff --git a/tests/requirements-mac.txt b/tests/requirements-mac.txt
--- a/tests/requirements-mac.txt
+++ b/tests/requirements-mac.txt
@@ -1,5 +1,17 @@
-# gevent wheels for mac os x cause pytest to crash on travis
+# OS X-specific test libraries.
+#
+# Libraries listed below are only installed by OS X-specific continuous
+# integration (CI) services (e.g., Travis). Platform-agnostic libraries are
+# listed in the "requirements-libraries.txt" file included below.
+
+# Prohibit "gevent" wheels, which cause py.test to crash on Travis under OS X.
 --no-binary gevent
+
+# Install all platform-agnostic libraries.
 -r requirements-libraries.txt
 
 pyenchant
+
+# SciPy currently provides no binary wheel for Windows and hence must be
+# isolated to non-Windows platforms. See also: scipy/scipy#5461
+scipy
diff --git a/tests/requirements-win.txt b/tests/requirements-win.txt
--- a/tests/requirements-win.txt
+++ b/tests/requirements-win.txt
@@ -1,4 +1,10 @@
-# Include some common dependencies for tests.
+# Windows-specific test libraries.
+#
+# Libraries listed below are only installed by Windows-specific continuous
+# integration (CI) services (e.g., AppVeyor). Platform-agnostic libraries are
+# listed in the "requirements-libraries.txt" file included below.
+
+# Install all platform-agnostic libraries.
 -r requirements-libraries.txt
 
 pypiwin32  # This package is a pip-installable version of PyWin32.
diff --git a/tests/requirements.txt b/tests/requirements.txt
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1 +1,5 @@
+# FIXME: Consider removing this file. All CI configuration files (e.g.,
+# ".travis.yml") only install from platform-specific requirement files (e.g.,
+# "requirements-linux.txt").
+
 -r requirements-libraries.txt