Repository: pyinstaller/pyinstaller

Issue:
Hook for 'phonenumbers.data.'
This program:

    import phonenumbers
    Number = "+17034820623"
    Parsed_Number = phonenumbers.parse(Number)
    print("The parsed number is <%s>" % (Parsed_Number))

works when invoked directly using Python 3.6.4:

    $ PF.py
    The parsed number is <Country Code: 1 National Number: 7034820623>

but when packaged with pyinstaller 3.4.dev0:

    $ pyinstaller --debug --onedir PF.py

fails, ultimately with:

    ModuleNotFoundError: No module named 'phonenumbers.data.region_US'

which the phonenumbers package appears to try to load dynamically. It's in:

    D:\Python36\Lib\site-packages\phonenumbers\data\region_US.py

That directory contains an "__init__.py" along with more than 300 other (small) modules which "phonenumbers" loads as needed.

Is there some magic to make this work with "--onedir", but ideally with "--onefile"?  To give the script another place to look for those modules? To incorporate them into the .EXE? Since I'm fairly new to Python and totally new to pyinstaller, I don't doubt I'm overlooking something. The recipe for pytz seems to hold some promise, but I can't be sure and I'm not yet knowledgeable enough to work with it.

When I try to attach pyinstaller's run log or the executable's, I'm being told "we can't process that file", so I'm pasting them here. First the executable's run log:

    $ dist\PF\PF.exe
    [4300] PyInstaller Bootloader 3.x
    [4300] LOADER: executable is f:\PFD\dist\PF\PF.exe
    [4300] LOADER: homepath is f:\PFD\dist\PF
    [4300] LOADER: _MEIPASS2 is NULL
    [4300] LOADER: archivename is f:\PFD\dist\PF\PF.exe
    [4300] LOADER: No need to extract files to run; setting extractionpath to homepath
    [4300] LOADER: SetDllDirectory(f:\PFD\dist\PF)
    [4300] LOADER: Already in the child - running user's code.
    [4300] LOADER: Python library: f:\PFD\dist\PF\python36.dll
    [4300] LOADER: Loaded functions from Python library.
    [4300] LOADER: Manipulating environment (sys.path, sys.prefix)
    [4300] LOADER: Pre-init sys.path is f:\PFD\dist\PF\base_library.zip;f:\PFD\dist\PF
    [4300] LOADER: sys.prefix is f:\PFD\dist\PF
    [4300] LOADER: Setting runtime options
    [4300] LOADER: Initializing python
    [4300] LOADER: Overriding Python's sys.path
    [4300] LOADER: Post-init sys.path is f:\PFD\dist\PF\base_library.zip;f:\PFD\dist\PF
    [4300] LOADER: Setting sys.argv
    [4300] LOADER: setting sys._MEIPASS
    [4300] LOADER: importing modules from CArchive
    [4300] LOADER: extracted struct
    [4300] LOADER: callfunction returned...
    [4300] LOADER: extracted pyimod01_os_path
    [4300] LOADER: callfunction returned...
    [4300] LOADER: extracted pyimod02_archive
    [4300] LOADER: callfunction returned...
    [4300] LOADER: extracted pyimod03_importers
    [4300] LOADER: callfunction returned...
    [4300] LOADER: Installing PYZ archive with Python modules.
    [4300] LOADER: PYZ archive: out00-PYZ.pyz
    [4300] LOADER: Running pyiboot01_bootstrap.py
    [4300] LOADER: Running PF.py
    [4300] Failed to execute script PF
    [4300] LOADER: OK.
    [4300] LOADER: Cleaning up Python interpreter.
    Traceback (most recent call last):
    File "PF.py", line 4, in <module>
    Parsed_Number = phonenumbers.parse(Number)
    File "site-packages\phonenumbers\phonenumberutil.py", line 2835, in parse
    File "site-packages\phonenumbers\phonemetadata.py", line 294, in metadata_for_region_or_calling_code
    File "site-packages\phonenumbers\phonemetadata.py", line 267, in metadata_for_region
    File "site-packages\phonenumbers\data\__init__.py", line 23, in _load_region
    ModuleNotFoundError: No module named 'phonenumbers.data.region_US'

And pyinstaller's run log:

    $ pyinstaller --debug --onedir PF.py
    50 INFO: PyInstaller: 3.4.dev0
    50 INFO: Python: 3.6.4
    50 INFO: Platform: Windows-7-6.1.7601-SP1
    50 INFO: wrote f:\PFD\PF.spec
    80 INFO: UPX is available.
    80 INFO: Extending PYTHONPATH with paths
    ['f:\\PFD', 'f:\\PFD']
    80 INFO: checking Analysis
    80 INFO: Building Analysis because out00-Analysis.toc is non existent
    80 INFO: Initializing module dependency graph...
    80 INFO: Initializing module graph hooks...
    80 INFO: Analyzing base_library.zip ...
    2320 INFO: running Analysis out00-Analysis.toc
    2330 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
    required by d:\python36\python.exe
    2790 INFO: Caching module hooks...
    2790 INFO: Analyzing f:\PFD\PF.py
    2960 INFO: Loading module hooks...
    2960 INFO: Loading module hook "hook-encodings.py"...
    3010 INFO: Loading module hook "hook-pydoc.py"...
    3010 INFO: Loading module hook "hook-xml.py"...
    3190 INFO: Looking for ctypes DLLs
    3190 INFO: Analyzing run-time hooks ...
    3190 INFO: Looking for dynamic libraries
    3250 INFO: Looking for eggs
    3250 INFO: Using Python library d:\python36\python36.dll
    3250 INFO: Found binding redirects:
    []
    3250 INFO: Warnings written to f:\PFD\build\PF\warnPF.txt
    3280 INFO: Graph cross-reference written to f:\PFD\build\PF\xref-PF.html
    3290 INFO: checking PYZ
    3290 INFO: Building PYZ because out00-PYZ.toc is non existent
    3290 INFO: Building PYZ (ZlibArchive) f:\PFD\build\PF\out00-PYZ.pyz
    3660 INFO: Building PYZ (ZlibArchive) f:\PFD\build\PF\out00-PYZ.pyz completed successfully.
    3660 INFO: checking PKG
    3660 INFO: Building PKG because out00-PKG.toc is non existent
    3660 INFO: Building PKG (CArchive) out00-PKG.pkg
    3670 INFO: Building PKG (CArchive) out00-PKG.pkg completed successfully.
    3670 INFO: Bootloader d:\python36\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run_d.exe
    3670 INFO: checking EXE
    3670 INFO: Building EXE because out00-EXE.toc is non existent
    3670 INFO: Building EXE from out00-EXE.toc
    3670 INFO: Appending archive to EXE f:\PFD\build\PF\PF.exe
    3670 INFO: Building EXE from out00-EXE.toc completed successfully.
    3670 INFO: checking COLLECT
    3670 INFO: Building COLLECT because out00-COLLECT.toc is non existent
    3670 INFO: Building COLLECT out00-COLLECT.toc
    3740 INFO: Building COLLECT out00-COLLECT.toc completed successfully.



Discussion:
I assume for full functionality you need to add all these 300 hidden modules. For this create a hook using `collect_submodules()` (see https://pyinstaller.readthedocs.io/en/stable/hooks.html#useful-items-in-pyinstaller-utils-hooks). See [this section of the manual](https://pyinstaller.readthedocs.org/en/latest/hooks.html) for more information about how to implement hooks. 

We would appreciate if you'd submit a pull-request for the hook for this then :-) See our Development Guide on [how to create pull-requests for PyInstaller](https://pyinstaller.readthedocs.io/en/stable/development/pull-requests.html). Thanks.

Hi @htgoebel!

I just submitted PR #3558 which implements this hook.

Could you please check it?

Thanks!

Resolving patch:
diff --git a/PyInstaller/hooks/hook-phonenumbers.py b/PyInstaller/hooks/hook-phonenumbers.py
new file mode 100644
--- /dev/null
+++ b/PyInstaller/hooks/hook-phonenumbers.py
@@ -0,0 +1,16 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2017, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License with exception
+# for distributing bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#-----------------------------------------------------------------------------
+#
+# Hook for the phonenumbers package: https://pypi.org/project/phonenumbers/
+#
+# Tested with phonenumbers 8.9.7 and Python 3.6.1, on Ubuntu 16.04 64bit.
+
+from PyInstaller.utils.hooks import collect_submodules
+
+hiddenimports = collect_submodules('phonenumbers')


Test patch:
diff --git a/tests/functional/test_libraries.py b/tests/functional/test_libraries.py
--- a/tests/functional/test_libraries.py
+++ b/tests/functional/test_libraries.py
@@ -794,3 +794,15 @@ def test_uvloop(pyi_builder):
 @importorskip('web3')
 def test_web3(pyi_builder):
     pyi_builder.test_source("import web3")
+
+@importorskip('phonenumbers')
+def test_phonenumbers(pyi_builder):
+    pyi_builder.test_source("""
+        import phonenumbers
+        
+        number = '+17034820623'
+        parsed_number = phonenumbers.parse(number)
+
+        assert(parsed_number.country_code == 1)
+        assert(parsed_number.national_number == 7034820623)
+        """)
diff --git a/tests/requirements-libraries.txt b/tests/requirements-libraries.txt
--- a/tests/requirements-libraries.txt
+++ b/tests/requirements-libraries.txt
@@ -46,6 +46,7 @@ Unidecode==1.0.22
 h5py==2.7.1
 matplotlib==2.2.2
 python-dateutil==2.7.3
+phonenumbers==8.9.7
 
 # Special cases
 # -------------