Repository: pyinstaller/pyinstaller

Issue:
Symbol not found PyCryptoDome Hash
App won't run when packaged from commandline.
I'm stumped by this. Below is the MWE (virtualenv, python script, spec).
This test was ran using the dev version of PyInstaller

```
$ pipenv run python crypto_test.py
9b71d224bd62f3785d96d46ad3ea3d73319bfbc2890caadae2dff72519673ca72323c3d99ba5c11d7c7acc6e14b8c5da0c4663475c2e5c3adef46f73bcdec043

$ ./dist/crypto_test/crypto_test
Traceback (most recent call last):
  File "crypto_test.py", line 4, in <module>
    c = SHA512.new("hello".encode())
  File "site-packages/Cryptodome/Hash/SHA512.py", line 177, in new
  File "site-packages/Cryptodome/Hash/SHA512.py", line 85, in __init__
  File "ctypes/__init__.py", line 361, in __getattr__
  File "ctypes/__init__.py", line 366, in __getitem__
AttributeError: dlsym(0x7fe0f2e10ee0, SHA512_init): symbol not found
[22551] Failed to execute script crypto_test
```

Python Version: 3.6.6
OS: macOS 10.13.6

virtualenv (output from `pip freeze`):
```
altgraph==0.16.1
future==0.17.1
macholib==1.11
pefile==2018.8.8
pycryptodomex==3.7.3
PyInstaller==3.5.dev0+0d289e49b
```

Script (`crypto_test.py`):
```python3
from Cryptodome.Hash import SHA512

if __name__ == "__main__":
    c = SHA512.new("hello".encode())
    print(c.hexdigest())
```

Spec:
```
# -*- mode: python ; coding: utf-8 -*-
block_cipher = None

a = Analysis(['crypto_test.py'],
             pathex=['/private/tmp/crypto_test'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='crypto_test',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=False,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=False,
               name='crypto_test')
```

Build Output
```
$ pipenv run pyinstaller crypto_test.spec --onedir --clean -y
82 INFO: PyInstaller: 3.5.dev0+0d289e49b
82 INFO: Python: 3.6.6
89 INFO: Platform: Darwin-17.7.0-x86_64-i386-64bit
96 INFO: UPX is available.
98 INFO: Removing temporary files and cleaning cache in /Users/huw/Library/Application Support/pyinstaller
123 INFO: Extending PYTHONPATH with paths
['/private/tmp/crypto_test', '/private/tmp/crypto_test']
123 INFO: checking Analysis
124 INFO: Building Analysis because Analysis-00.toc is non existent
124 INFO: Initializing module dependency graph...
126 INFO: Initializing module graph hooks...
128 INFO: Analyzing base_library.zip ...
3239 INFO: running Analysis Analysis-00.toc
3248 INFO: Caching module hooks...
3252 INFO: Analyzing crypto_test.py
3346 INFO: Loading module hooks...
3346 INFO: Loading module hook "hook-encodings.py"...
3449 INFO: Loading module hook "hook-xml.py"...
3632 INFO: Loading module hook "hook-pydoc.py"...
3632 INFO: Loading module hook "hook-Cryptodome.py"...
3648 INFO: Looking for ctypes DLLs
3652 INFO: Analyzing run-time hooks ...
3657 INFO: Looking for dynamic libraries
3801 INFO: Looking for eggs
3801 INFO: Using Python library /Users/huw/.pyenv/versions/3.6.6/lib/libpython3.6m.dylib
3805 INFO: Warnings written to /private/tmp/crypto_test/build/crypto_test/warn-crypto_test.txt
3827 INFO: Graph cross-reference written to /private/tmp/crypto_test/build/crypto_test/xref-crypto_test.html
3833 INFO: checking PYZ
3833 INFO: Building PYZ because PYZ-00.toc is non existent
3833 INFO: Building PYZ (ZlibArchive) /private/tmp/crypto_test/build/crypto_test/PYZ-00.pyz
4090 INFO: Building PYZ (ZlibArchive) /private/tmp/crypto_test/build/crypto_test/PYZ-00.pyz completed successfully.
4093 INFO: checking PKG
4093 INFO: Building PKG because PKG-00.toc is non existent
4093 INFO: Building PKG (CArchive) PKG-00.pkg
4097 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
4098 INFO: Bootloader /Users/huw/.local/share/virtualenvs/crypto_test-nR4wJhA5/lib/python3.6/site-packages/PyInstaller/bootloader/Darwin-64bit/run
4098 INFO: checking EXE
4098 INFO: Building EXE because EXE-00.toc is non existent
4098 INFO: Building EXE from EXE-00.toc
4098 INFO: Appending archive to EXE /private/tmp/crypto_test/build/crypto_test/crypto_test
4099 INFO: Fixing EXE for code signing /private/tmp/crypto_test/build/crypto_test/crypto_test
4103 INFO: Building EXE from EXE-00.toc completed successfully.
4103 INFO: checking COLLECT
4103 INFO: Building COLLECT because COLLECT-00.toc is non existent
4104 INFO: Removing dir /private/tmp/crypto_test/dist/crypto_test
4116 INFO: Building COLLECT COLLECT-00.toc
5417 INFO: Building COLLECT COLLECT-00.toc completed successfully.
```


Discussion:
~Submitted without giving all the info by accident, will update~
For my app, I've switched from PyCryptodome to cryptography and this works

I suspect it worked because when I build the app with cryptography PyInstaller loaded the `cryptography` but I don't see that in the build output above.

If this is the reason, I'm happy to close the issue
I have similar problem on macOS (but not on Windows). I can find the symbol _SHA256_init (with underscore) in Crypto/Hash/_SHA256.cpython-37m-darwin.so
```
nm -C _SHA256.cpython-37m-darwin.so
0000000000002dc0 s _H
0000000000002810 T _SHA256_copy
0000000000000750 T _SHA256_destroy
00000000000025e0 T _SHA256_digest
00000000000006e0 T _SHA256_init
0000000000002840 T _SHA256_pbkdf2_hmac_assist
0000000000000760 T _SHA256_update
...
```

On my linux box I have symbols without underscore - that's what pycryptodome expects to find:
```
nm -C _SHA256.cpython-38-x86_64-linux-gnu.so
0000000000002bb0 T SHA256_copy
00000000000029a0 T SHA256_destroy
0000000000002ae0 T SHA256_digest
0000000000002910 T SHA256_init
0000000000002c50 T SHA256_pbkdf2_hmac_assist
00000000000029b0 T SHA256_update
...
```

Also look here https://github.com/Legrandin/pycryptodome/issues/281
Re-igniting an old issue, switching to pycryptodomex rather than pycryptodome seems to resolve the issue for me.

Then changing imports from:

`from Crypto.x` to `from Pycrypto.x` 

May be a fluke but works in my situation.

Resolving patch:
diff --git a/PyInstaller/building/api.py b/PyInstaller/building/api.py
--- a/PyInstaller/building/api.py
+++ b/PyInstaller/building/api.py
@@ -840,7 +840,11 @@ def _set_dependencies(self, analysis, path):
                     # relative path needs to be reconstructed from the
                     # name components.
                     if tpl[2] == 'EXTENSION':
-                        ext_components = tpl[0].split('.')[:-1]
+                        # Split on os.path.sep first, to handle additional
+                        # path prefix (e.g., lib-dynload)
+                        ext_components = tpl[0].split(os.path.sep)
+                        ext_components = ext_components[:-1] \
+                            + ext_components[-1].split('.')[:-1]
                         if ext_components:
                             rel_path = os.path.join(*ext_components)
                         else:
diff --git a/PyInstaller/building/build_main.py b/PyInstaller/building/build_main.py
--- a/PyInstaller/building/build_main.py
+++ b/PyInstaller/building/build_main.py
@@ -480,6 +480,18 @@ def assemble(self):
             self.binding_redirects[:] = list(set(self.binding_redirects))
             logger.info("Found binding redirects: \n%s", self.binding_redirects)
 
+        # Filter binaries to adjust path of extensions that come from
+        # python's lib-dynload directory. Prefix them with lib-dynload
+        # so that we'll collect them into subdirectory instead of
+        # directly into _MEIPASS
+        for idx, tpl in enumerate(self.binaries):
+            name, path, typecode = tpl
+            if typecode == 'EXTENSION' \
+               and not os.path.dirname(os.path.normpath(name)) \
+               and os.path.basename(os.path.dirname(path)) == 'lib-dynload':
+                name = os.path.join('lib-dynload', name)
+                self.binaries[idx] = (name, path, typecode)
+
         # Place Python source in data files for the noarchive case.
         if self.noarchive:
             # Create a new TOC of ``(dest path for .pyc, source for .py, type)``.


Test patch:
diff --git a/tests/functional/test_import.py b/tests/functional/test_import.py
--- a/tests/functional/test_import.py
+++ b/tests/functional/test_import.py
@@ -11,6 +11,7 @@
 #-----------------------------------------------------------------------------
 
 import os
+import sys
 import glob
 import ctypes
 import ctypes.util
@@ -371,6 +372,31 @@ def g():
     pyi_builder.test_source(source % locals(), test_id=test_id)
 
 
+def test_ctypes_cdll_builtin_extension(pyi_builder):
+    # Take a built-in that is provided as an extension
+    builtin_ext = '_sha256'
+    if builtin_ext in sys.builtin_module_names:
+        # On Windows, built-ins do not seem to be extensions
+        pytest.skip(f"{builtin_ext} is a built-in module without extension.")
+
+    pyi_builder.test_source(
+        """
+        import ctypes
+        import importlib.machinery
+
+        # Try to load CDLL with all possible extension suffices; this
+        # should fail in all cases, as built-in extensions should not
+        # be in the ctypes' search path.
+        builtin_ext = '{0}'
+        for suffix in importlib.machinery.EXTENSION_SUFFIXES:
+            try:
+                lib = ctypes.CDLL(builtin_ext + suffix)
+            except OSError:
+                lib = None
+            assert lib is None, "Built-in extension picked up by ctypes.CDLL!"
+        """.format(builtin_ext))
+
+
 # TODO: Add test-cases for the prefabricated library loaders supporting
 # attribute accesses on windows. Example::
 #