Repository: pyinstaller/pyinstaller

Issue:
Recursion error with Pydantic
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the mailing list, see
	  <http://www.pyinstaller.org/support.html#mailing-list>
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

+++ ONLY TEXT +++ DO NOT POST IMAGES +++

## Description of the issue
Trying to run pyinstaller against a module that contains just `import pydantic` fails due to `RecursionError: maximum recursion depth exceeded while calling a Python object`.

## Context information (for bug reports)

* Output of `pyinstaller --version`: ```4.0```
* Version of Python: 3.8
* Platform: Fedora 32
* Did you also try this on another platform? Does it work there? I didn't


* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [ ] start with clean installation
  * [X] use the latest development version
  * [ ] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [ ] Package your program in **--onedir mode**
  * [ ] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [ ] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

```
import pydantic
```

### Stacktrace / full error message

```
...
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2088, in _load_module
    m = self._load_package(fqname, pathname, packagepath)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2907, in _load_package
    self._load_module(fqname, fp, buf, stuff)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2088, in _load_module
    m = self._load_package(fqname, pathname, packagepath)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2907, in _load_package
    self._load_module(fqname, fp, buf, stuff)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2088, in _load_module
    m = self._load_package(fqname, pathname, packagepath)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2907, in _load_package
    self._load_module(fqname, fp, buf, stuff)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2088, in _load_module
    m = self._load_package(fqname, pathname, packagepath)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2880, in _load_package
    ns_pkgpath = _namespace_package_path(fqname, pkgpath or [], self.path)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 164, in _namespace_package_path
    working_set = pkg_resources.WorkingSet(path)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 567, in __init__
    self.add_entry(entry)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 623, in add_entry
    for dist in find_distributions(entry, True):
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2065, in find_on_path
    for dist in factory(fullpath):
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2134, in distributions_from_metadata
    yield Distribution.from_location(
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2590, in from_location
    return cls(
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2994, in _reload_version
    md_version = self._get_version()
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2772, in _get_version
    version = _version_from_file(lines)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2556, in _version_from_file
    line = next(iter(version_lines), '')
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2767, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1432, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1419, in get_metadata
    path = self._get_metadata_path(name)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1407, in _get_metadata_path
    return self._fn(self.egg_info, name)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1487, in _fn
    self._validate_resource_path(resource_name)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1547, in _validate_resource_path
    posixpath.isabs(path) or
  File "/usr/lib64/python3.8/posixpath.py", line 63, in isabs
    sep = _get_sep(s)
  File "/usr/lib64/python3.8/posixpath.py", line 42, in _get_sep
    if isinstance(path, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object
```

Please also see <https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs>
for more about what would use to solve the issue.



Discussion:
This may be caused by pydantic having two files for the same module with the same base name. I'm investigating.
Here's when things start going wrong:
```
2297 TRACE: safe_import_module 'test' 'test' None
2297 TRACE: _find_module_path <- 'test' ['/home/alex/workspace/cunit/venv/bin', '/usr/lib64/python38.zip', '/usr/lib64/python3.8', '/usr/lib64/python3.8/lib-dynload', '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages', '/home/alex/workspace/cunit/venv/lib/python3.8/site-packages', '/home/alex/workspace/cunit', '/home/alex/workspace/cunit']
2297 TRACE: _find_module_path -> (<_io.StringIO object at 0x7f7c292d7f70>, '/home/alex/workspace/cunit/test.py', ('.py', 'r', 1))
2297 TRACE: load_module 'test' 'fp' '/home/alex/workspace/cunit/test.py'
2297 TRACE: addNode SourceModule('test',)
2297 TRACE: _safe_import_hook 'pydantic' SourceModule('test', '/home/alex/workspace/cunit/test.py') None 0
2297 TRACE: _import_hook 'pydantic' SourceModule('test', '/home/alex/workspace/cunit/test.py') SourceModule('test', '/home/alex/workspace/cunit/test.py') 0
2297 TRACE: determine_parent SourceModule('test', '/home/alex/workspace/cunit/test.py')
2297 TRACE: determine_parent -> None
2297 TRACE: find_head_package None 'pydantic' 0
2297 TRACE: safe_import_module 'pydantic' 'pydantic' None
2297 TRACE: _find_module_path <- 'pydantic' ['/home/alex/workspace/cunit/venv/bin', '/usr/lib64/python38.zip', '/usr/lib64/python3.8', '/usr/lib64/python3.8/lib-dynload', '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages', '/home/alex/workspace/cunit/venv/lib/python3.8/site-packages', '/home/alex/workspace/cunit', '/home/alex/workspace/cunit']
2297 TRACE: _find_module_path -> (None, '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so', ('', '', 5))
2297 TRACE: load_module 'pydantic' None '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so'
2297 TRACE: load_package 'pydantic' '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so' []
2312 TRACE: addNode Package('pydantic',)
2312 TRACE: find __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2312 TRACE: _find_module_path <- 'pydantic.__init__' ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic']
2312 TRACE: _find_module_path -> (None, '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so', ('', '', 5))
2312 TRACE: load __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2312 TRACE: load_module 'pydantic' None '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so'
2313 TRACE: load_package 'pydantic' '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so' []
2327 TRACE: find __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2327 TRACE: _find_module_path <- 'pydantic.__init__' ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic']
2327 TRACE: _find_module_path -> (None, '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so', ('', '', 5))
2327 TRACE: load __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2327 TRACE: load_module 'pydantic' None '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so'
2327 TRACE: load_package 'pydantic' '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so' []
2342 TRACE: find __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2342 TRACE: _find_module_path <- 'pydantic.__init__' ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic']
2342 TRACE: _find_module_path -> (None, '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so', ('', '', 5))
...
```
Looks like this is a duplicate of https://github.com/pyinstaller/pyinstaller/issues/4346.

Resolving patch:
diff --git a/PyInstaller/compat.py b/PyInstaller/compat.py
--- a/PyInstaller/compat.py
+++ b/PyInstaller/compat.py
@@ -652,6 +652,7 @@ def importlib_load_source(name, pathname):
 # dependency graph.
 BINARY_MODULE_TYPES = {
     'Extension',
+    'ExtensionPackage',
 }
 # Object types of valid Python modules in modulegraph dependency graph.
 VALID_MODULE_TYPES = PURE_PYTHON_MODULE_TYPES | SPECIAL_MODULE_TYPES | BINARY_MODULE_TYPES
@@ -686,6 +687,7 @@ def importlib_load_source(name, pathname):
     'ArchiveModule': 'PYMODULE',
     # Binary modules.
     'Extension': 'EXTENSION',
+    'ExtensionPackage': 'EXTENSION',
     # Special valid modules.
     'BuiltinModule': 'BUILTIN',
     'NamespacePackage': 'PYMODULE',
diff --git a/PyInstaller/depend/analysis.py b/PyInstaller/depend/analysis.py
--- a/PyInstaller/depend/analysis.py
+++ b/PyInstaller/depend/analysis.py
@@ -27,6 +27,8 @@
  Package          basename         full path to __init__.py
         packagepath is ['path to package']
         globalnames is set of global names __init__.py defines
+ ExtensionPackage basename         full path to __init__.{so,dll}
+        packagepath is ['path to package']
 
 The main extension here over ModuleGraph is a method to extract nodes
 from the flattened graph and return them as a TOC, or added to a TOC.
@@ -533,42 +535,14 @@ def _make_toc(self, typecode=None, existing_TOC=None):
 
         result = existing_TOC or TOC()
         for node in self.flatten(start=self._top_script_node):
-            # TODO This is terrible. Everything in Python has a type. It's
-            # nonsensical to even speak of "nodes [that] are not typed." How
-            # would that even occur? After all, even "None" has a type! (It's
-            # "NoneType", for the curious.) Remove this, please.
-
             # Skip modules that are in base_library.zip.
             if module_filter.match(node.identifier):
                 continue
-
-            # get node type e.g. Script
-            mg_type = type(node).__name__
-            assert mg_type is not None
-
-            if typecode and not (mg_type in typecode):
-                # Type is not a to be selected one, skip this one
-                continue
-            # Extract the identifier and a path if any.
-            if mg_type == 'Script':
-                # for Script nodes only, identifier is a whole path
-                (name, ext) = os.path.splitext(node.filename)
-                name = os.path.basename(name)
-            else:
-                name = node.identifier
-            path = node.filename if node.filename is not None else ''
-            # Ensure name is really 'str'. Module graph might return
-            # object type 'modulegraph.Alias' which inherits fromm 'str'.
-            # But 'marshal.dumps()' function is able to marshal only 'str'.
-            # Otherwise on Windows PyInstaller might fail with message like:
-            #
-            #   ValueError: unmarshallable object
-            name = str(name)
-            # Translate to the corresponding TOC typecode.
-            toc_type = MODULE_TYPES_TO_TOC_DICT[mg_type]
-            # TOC.append the data. This checks for a pre-existing name
-            # and skips it if it exists.
-            result.append((name, path, toc_type))
+            entry = self._node_to_toc(node, typecode)
+            if entry is not None:
+                # TOC.append the data. This checks for a pre-existing name
+                # and skips it if it exists.
+                result.append(entry)
         return result
 
     def make_pure_toc(self):
@@ -590,6 +564,46 @@ def make_missing_toc(self):
         """
         return self._make_toc(BAD_MODULE_TYPES)
 
+    @staticmethod
+    def _node_to_toc(node, typecode=None):
+        # TODO This is terrible. Everything in Python has a type. It's
+        # nonsensical to even speak of "nodes [that] are not typed." How
+        # would that even occur? After all, even "None" has a type! (It's
+        # "NoneType", for the curious.) Remove this, please.
+
+        # get node type e.g. Script
+        mg_type = type(node).__name__
+        assert mg_type is not None
+
+        if typecode and not (mg_type in typecode):
+            # Type is not a to be selected one, skip this one
+            return None
+        # Extract the identifier and a path if any.
+        if mg_type == 'Script':
+            # for Script nodes only, identifier is a whole path
+            (name, ext) = os.path.splitext(node.filename)
+            name = os.path.basename(name)
+        elif mg_type == 'ExtensionPackage':
+            # package with __init__ module being an extension module
+            # This needs to end up as e.g. 'mypkg/__init__.so'.
+            # Convert the packages name ('mypkg') into the module name
+            # ('mypkg.__init__') *here* to keep special cases away elsewhere
+            # (where the module name is converted to a filename).
+            name = node.identifier + ".__init__"
+        else:
+            name = node.identifier
+        path = node.filename if node.filename is not None else ''
+        # Ensure name is really 'str'. Module graph might return
+        # object type 'modulegraph.Alias' which inherits fromm 'str'.
+        # But 'marshal.dumps()' function is able to marshal only 'str'.
+        # Otherwise on Windows PyInstaller might fail with message like:
+        #
+        #   ValueError: unmarshallable object
+        name = str(name)
+        # Translate to the corresponding TOC typecode.
+        toc_type = MODULE_TYPES_TO_TOC_DICT[mg_type]
+        return (name, path, toc_type)
+
     def nodes_to_toc(self, node_list, existing_TOC=None):
         """
         Given a list of nodes, create a TOC representing those nodes.
@@ -600,15 +614,7 @@ def nodes_to_toc(self, node_list, existing_TOC=None):
         """
         result = existing_TOC or TOC()
         for node in node_list:
-            mg_type = type(node).__name__
-            toc_type = MODULE_TYPES_TO_TOC_DICT[mg_type]
-            if mg_type == "Script" :
-                (name, ext) = os.path.splitext(node.filename)
-                name = os.path.basename(name)
-            else:
-                name = node.identifier
-            path = node.filename if node.filename is not None else ''
-            result.append( (name, path, toc_type) )
+            result.append(self._node_to_toc(node))
         return result
 
     # Return true if the named item is in the graph as a BuiltinModule node.
diff --git a/PyInstaller/lib/modulegraph/modulegraph.py b/PyInstaller/lib/modulegraph/modulegraph.py
--- a/PyInstaller/lib/modulegraph/modulegraph.py
+++ b/PyInstaller/lib/modulegraph/modulegraph.py
@@ -37,6 +37,19 @@
 BOM = codecs.BOM_UTF8.decode('utf-8')
 
 
+class BUILTIN_MODULE:
+    def is_package(fqname):
+        return False
+
+
+class NAMESPACE_PACKAGE:
+    def __init__(self, namespace_dirs):
+        self.namespace_dirs = namespace_dirs
+
+    def is_package(self, fqname):
+        return True
+
+
 #FIXME: Leverage this rather than magic numbers below.
 ABSOLUTE_OR_RELATIVE_IMPORT_LEVEL = -1
 """
@@ -834,6 +847,14 @@ class Package(BaseModule):
     pass
 
 
+class ExtensionPackage(Extension, Package):
+    """
+    Graph node representing a package where the __init__ module is an extension
+    module.
+    """
+    pass
+
+
 class NamespacePackage(Package):
     """
     Graph node representing a namespace package.
@@ -2024,9 +2045,6 @@ def _safe_import_module(
             # this module. This effectively defaults to "sys.path".
             search_dirs = None
 
-            # Open file handle providing the physical contents of this module.
-            file_handle = None
-
             # If this module has a parent package...
             if parent_module is not None:
                 # ...with a list of the absolute paths of all directories
@@ -2039,18 +2057,13 @@ def _safe_import_module(
                     return None
 
             try:
-                try:
-                    file_handle, pathname, metadata = self._find_module(
-                        module_partname, search_dirs, parent_module)
-                except ImportError as exc:
-                    self.msgout(3, "safe_import_module -> None (%r)" % exc)
-                    return None
+                pathname, loader = self._find_module(
+                    module_partname, search_dirs, parent_module)
+            except ImportError as exc:
+                self.msgout(3, "safe_import_module -> None (%r)" % exc)
+                return None
 
-                module = self._load_module(
-                    module_name, file_handle, pathname, metadata)
-            finally:
-                if file_handle is not None:
-                    file_handle.close()
+            module = self._load_module(module_name, pathname, loader)
 
         # If this is a submodule rather than top-level module...
         if parent_module is not None:
@@ -2072,76 +2085,83 @@ def _safe_import_module(
         self.msgout(3, "safe_import_module ->", module)
         return module
 
+    def _load_module(self, fqname, pathname, loader):
+        from importlib._bootstrap_external import ExtensionFileLoader
+        self.msgin(2, "load_module", fqname, pathname,
+                   loader.__class__.__name__)
+        partname = fqname.rpartition(".")[-1]
 
-    #FIXME: For clarity, rename method parameters to:
-    #    def _load_module(self, module_name, file_handle, pathname, imp_info):
-    def _load_module(self, fqname, fp, pathname, info):
-        suffix, mode, typ = info
-        self.msgin(2, "load_module", fqname, fp and "fp", pathname)
-
-        if typ == imp.PKG_DIRECTORY:
-            if isinstance(mode, (list, tuple)):
-                packagepath = mode
+        if loader.is_package(partname):
+            if isinstance(loader, NAMESPACE_PACKAGE):
+                pkgpath = loader.namespace_dirs[:]  # copy for safety
             else:
-                packagepath = []
-
-            m = self._load_package(fqname, pathname, packagepath)
-            self.msgout(2, "load_module ->", m)
-            return m
-
-        if typ == imp.PY_SOURCE:
-            contents = fp.read()
-            if isinstance(contents, bytes):
-                contents += b'\n'
-            else:
-                contents += '\n'
-
-            try:
-                co = compile(contents, pathname, 'exec', ast.PyCF_ONLY_AST, True)
-                if sys.version_info[:2] == (3, 5):
-                    # In Python 3.5 some syntax problems with async
-                    # functions are only reported when compiling to bytecode
-                    compile(co, '-', 'exec', 0, True)
-            except SyntaxError:
-                co = None
-                cls = InvalidSourceModule
-                self.msg(2, "load_module: InvalidSourceModule", pathname)
-
-            else:
-                cls = SourceModule
-
-        elif typ == imp.PY_COMPILED:
-            data = fp.read(4)
-            magic = imp.get_magic()
-            if data != magic:
-                self.msg(2, "load_module: InvalidCompiledModule, "
-                         "bad magic number", pathname, data, magic)
-                co = None
-                cls = InvalidCompiledModule
+                pkgpath = []
+
+            newname = _replacePackageMap.get(fqname)
+            if newname:
+                fqname = newname
+            ns_pkgpath = _namespace_package_path(
+                fqname, pkgpath or [], self.path)
+
+            if ns_pkgpath or pkgpath:
+                # this is a namespace package
+                m = self.createNode(NamespacePackage, fqname)
+                m.filename = '-'
+                m.packagepath = ns_pkgpath
             else:
-                if sys.version_info >= (3, 7):
-                    fp.read(12)
-                elif sys.version_info >= (3, 4):
-                    fp.read(8)
+                if isinstance(loader, ExtensionFileLoader):
+                    m = self.createNode(ExtensionPackage, fqname)
                 else:
-                    fp.read(4)
+                    m = self.createNode(Package, fqname)
+                m.filename = pathname
+                # PEP-302-compliant loaders return the pathname of the
+                # `__init__`-file, not the packge directory.
+                assert os.path.basename(pathname).startswith('__init__.')
+                m.packagepath = [os.path.dirname(pathname)] + ns_pkgpath
+
+            # As per comment at top of file, simulate runtime packagepath
+            # additions
+            m.packagepath = m.packagepath + self._package_path_map.get(
+                fqname, [])
+
+            if isinstance(m, NamespacePackage):
+                return m
+
+        co = None
+        if loader is BUILTIN_MODULE:
+            cls = BuiltinModule
+        elif isinstance(loader, ExtensionFileLoader):
+            cls = Extension
+        else:
+            src = loader.get_source(partname)
+            if src is not None:
                 try:
-                    co = marshal.loads(fp.read())
-                    cls = CompiledModule
-                except Exception as exc:
+                    co = compile(src, pathname, 'exec', ast.PyCF_ONLY_AST,
+                                 True)
+                    cls = SourceModule
+                    if sys.version_info[:2] == (3, 5):
+                        # In Python 3.5 some syntax problems with async
+                        # functions are only reported when compiling to
+                        # bytecode
+                        compile(co, '-', 'exec', 0, True)
+                except SyntaxError:
+                    co = None
+                    cls = InvalidSourceModule
+                except Exception as exc:  # FIXME: more specific?
+                    cls = InvalidSourceModule
+                    self.msg(2, "load_module: InvalidSourceModule", pathname,
+                             exc)
+            else:
+                # no src available
+                try:
+                    co = loader.get_code(partname)
+                    cls = (CompiledModule if co is not None
+                           else InvalidCompiledModule)
+                except Exception as exc:  # FIXME: more specific?
                     self.msg(2, "load_module: InvalidCompiledModule, "
                              "Cannot load code", pathname, exc)
-                    co = None
                     cls = InvalidCompiledModule
 
-        elif typ == imp.C_BUILTIN:
-            cls = BuiltinModule
-            co = None
-
-        else:
-            cls = Extension
-            co = None
-
         m = self.createNode(cls, fqname)
         m.filename = pathname
         if co is not None:
@@ -2868,50 +2888,6 @@ def _process_imports(self, source_module):
         source_module._deferred_imports = None
 
 
-    def _load_package(self, fqname, pathname, pkgpath):
-        """
-        Called only when an imp.PKG_DIRECTORY is found
-        """
-        self.msgin(2, "load_package", fqname, pathname, pkgpath)
-        newname = _replacePackageMap.get(fqname)
-        if newname:
-            fqname = newname
-
-        ns_pkgpath = _namespace_package_path(fqname, pkgpath or [], self.path)
-        if ns_pkgpath or pkgpath:
-            # this is a namespace package
-            m = self.createNode(NamespacePackage, fqname)
-            m.filename = '-'
-            m.packagepath = ns_pkgpath
-        else:
-            m = self.createNode(Package, fqname)
-            m.filename = pathname
-            # PEP-302-compliant loaders return the pathname of the
-            # `__init__`-file, not the packge directory.
-            if os.path.basename(pathname).startswith('__init__.'):
-                pathname = os.path.dirname(pathname)
-            m.packagepath = [pathname] + ns_pkgpath
-
-        # As per comment at top of file, simulate runtime packagepath additions.
-        m.packagepath = m.packagepath + self._package_path_map.get(fqname, [])
-
-        try:
-            self.msg(2, "find __init__ for %s"%(m.packagepath,))
-            fp, buf, stuff = self._find_module("__init__", m.packagepath, parent=m)
-        except ImportError:
-            pass
-
-        else:
-            try:
-                self.msg(2, "load __init__ for %s"%(m.packagepath,))
-                self._load_module(fqname, fp, buf, stuff)
-            finally:
-                if fp is not None:
-                    fp.close()
-        self.msgout(2, "load_package ->", m)
-        return m
-
-
     def _find_module(self, name, path, parent=None):
         """
         3-tuple describing the physical location of the module with the passed
@@ -2934,7 +2910,7 @@ def _find_module(self, name, path, parent=None):
 
         Returns
         ----------
-        (file_handle, filename, metadata)
+        (filename, loader)
             See `modulegraph._find_module()` for details.
 
         Raises
@@ -2956,7 +2932,7 @@ def _find_module(self, name, path, parent=None):
 
         if path is None:
             if name in sys.builtin_module_names:
-                return (None, None, ("", "", imp.C_BUILTIN))
+                return (None, BUILTIN_MODULE)
 
             path = self.path
 
@@ -2990,16 +2966,12 @@ def _find_module_path(self, fullname, module_name, search_dirs):
 
         Returns
         ----------
-        (file_handle, filename, metadata)
-            3-tuple describing the physical location of this module, where:
-            * `file_handle` is an open read-only file handle from which the
-              on-disk contents of this module may be read if this is a
-              pure-Python module or `None` otherwise (e.g., if this is a
-              package or C extension).
+        (filename, loader)
+            2-tuple describing the physical location of this module, where:
             * `filename` is the absolute path of this file.
-            * `metadata` is itself a 3-tuple `(filetype, open_mode, imp_type)`.
-              See the `_IMPORTABLE_FILETYPE_TO_METADATA` dictionary for
-              details.
+            * `loader` is the import loader.
+              In case of a namespace package, this is a NAMESPACE_PACKAGE
+              instance
 
         Raises
         ----------
@@ -3008,19 +2980,9 @@ def _find_module_path(self, fullname, module_name, search_dirs):
         """
         self.msgin(4, "_find_module_path <-", fullname, search_dirs)
 
-        # TODO: Under:
-        #
-        # * Python 3.3, the following logic should be replaced by logic
-        #   leveraging only the "importlib" module.
-        # * Python 3.4, the following logic should be replaced by a call to
-        #   importlib.util.find_spec().
-
-        # Top-level 3-tuple to be returned.
+        # Top-level 2-tuple to be returned.
         path_data = None
 
-        # File handle to be returned.
-        file_handle = None
-
         # List of the absolute paths of all directories comprising the
         # namespace package to which this module belongs if any.
         namespace_dirs = []
@@ -3058,9 +3020,6 @@ def _find_module_path(self, fullname, module_name, search_dirs):
                     # self.msg(4, "_find_module_path loader not found", search_dir)
                     continue
 
-                # 3-tuple of metadata to be returned.
-                metadata = None
-
                 # Absolute path of this module. If this module resides in a
                 # compressed archive, this is the absolute path of this module
                 # after extracting this module from that archive and hence
@@ -3086,78 +3045,14 @@ def _find_module_path(self, fullname, module_name, search_dirs):
                     self.msg(4, "_find_module_path path not found", pathname)
                     continue
 
-                # If this loader defines the PEP 302-compliant is_package()
-                # method returning True, this is a non-namespace package.
-                if hasattr(loader, 'is_package') and loader.is_package(module_name):
-                    metadata = ('', '', imp.PKG_DIRECTORY)
-                # Else, this is either a module or C extension.
-                else:
-                    # In either case, this path must have a filetype.
-                    # os.path.splitext won't work here since we sometimes need
-                    # to match more than just the file extension.
-                    filetype = [filetype
-                                for filetype in _IMPORTABLE_FILETYPE_EXTS
-                                if pathname.endswith(filetype)]
-                    if filetype:
-                        # at least one extension matched,
-                        # pick the first (longest) one
-                        filetype = filetype[0]
-                    else:
-                        raise ImportError(
-                            'Non-package module %r path %r has no filetype' % (module_name, pathname))
-
-                    # 3-tuple of metadata specific to this filetype.
-                    metadata = _IMPORTABLE_FILETYPE_TO_METADATA.get(
-                        filetype, None)
-                    if metadata is None:
-                        raise ImportError(
-                            'Non-package module %r filetype %r unrecognized' % (pathname, filetype))
-
-                    # See "_IMPORTABLE_FILETYPE_TO_METADATA" for details.
-                    open_mode = metadata[1]
-                    imp_type = metadata[2]
-
-                    # If this is a C extension, leave this path unopened.
-                    if imp_type == imp.C_EXTENSION:
-                        pass
-                    # Else, this is a module.
-                    #
-                    # If this loader defines the PEP 302-compliant get_source()
-                    # method, open the returned string as a file-like buffer.
-                    elif imp_type == imp.PY_SOURCE and hasattr(loader, 'get_source'):
-                        file_handle = StringIO(loader.get_source(module_name))
-                    # If this loader defines the PEP 302-compliant get_code()
-                    # method, open the returned object as a file-like buffer.
-                    elif imp_type == imp.PY_COMPILED and hasattr(loader, 'get_code'):
-                        try:
-                            code_object = loader.get_code(module_name)
-                            if code_object is None:
-                                file_handle = BytesIO(b'\0\0\0\0\0\0\0\0')
-                            else:
-                                file_handle = _code_to_file(code_object)
-                        except ImportError:
-                            # post-bone the ImportError until load_module
-                            file_handle = BytesIO(b'\0\0\0\0\0\0\0\0')
-                    # If this is an uncompiled file under Python 3, open this
-                    # file for encoding-aware text reading.
-                    elif imp_type == imp.PY_SOURCE and sys.version_info[0] == 3:
-                        with open(pathname, 'rb') as file_handle:
-                            encoding = util.guess_encoding(file_handle)
-                        file_handle = open(
-                            pathname, open_mode, encoding=encoding)
-                    # Else, this is either a compiled file or an uncompiled
-                    # file under Python 2. In either case, open this file.
-                    else:
-                        file_handle = open(pathname, open_mode)
-
                 # Return such metadata.
-                path_data = (file_handle, pathname, metadata)
+                path_data = (pathname, loader)
                 break
             # Else if this is a namespace package, return such metadata.
             else:
                 if namespace_dirs:
-                    path_data = (None, namespace_dirs[0], (
-                        '', namespace_dirs, imp.PKG_DIRECTORY))
+                    path_data = (namespace_dirs[0],
+                                 NAMESPACE_PACKAGE(namespace_dirs))
         except UnicodeDecodeError as exc:
             self.msgout(1, "_find_module_path -> unicode error", exc)
         # Ensure that exceptions are logged, as this function is typically


Test patch:
diff --git a/tests/functional/test_regression.py b/tests/functional/test_regression.py
--- a/tests/functional/test_regression.py
+++ b/tests/functional/test_regression.py
@@ -9,9 +9,11 @@
 # SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
 #-----------------------------------------------------------------------------
 
+from importlib.machinery import EXTENSION_SUFFIXES
 
-from PyInstaller.depend import analysis
+from PyInstaller.depend import analysis, bindepend
 from PyInstaller.building.build_main import Analysis
+from PyInstaller.building.api import PYZ
 
 def test_issue_2492(monkeypatch, tmpdir):
     # Crash if an extension module has an hidden import to ctypes (e.g. added
@@ -35,3 +37,46 @@ def test_issue_2492(monkeypatch, tmpdir):
     tmpdir.join('hook-_struct.py').write('hiddenimports = ["ctypes"]')
     a = Analysis([str(script)], hookspath=[str(tmpdir)],
                  excludes=['encodings', 'pydoc', 'xml', 'distutils'])
+
+
+def test_issue_5131(monkeypatch, tmpdir):
+    """
+    While fixing the endless recursion when the package's __init__ module is
+    an extension (see
+    tests/unit/test_modulegraph_more.py::package_init_is_extension_*), another
+    error occured: PyInstaller.building._utils._load_code() tried to complote
+    the source code for extension module - triggered by PYZ.assemble(), which
+    is collecting all source files - caused by this being marked as "PYMODULE"
+    in the TOC.
+    """
+
+    def getImports(*args, **kwargs):
+        # Our faked binary does not match the expected file-format for all
+        # platforms, thus the resp. code might crash. Simply ignore this.
+        try:
+            return orig_getImports(*args, **kwargs)
+        except:  # noqa
+            return []
+
+    monkeypatch.setattr('PyInstaller.config.CONF',
+                        {'workpath': str(tmpdir),
+                         'spec': str(tmpdir),
+                         'warnfile': str(tmpdir.join('warn.txt')),
+                         'dot-file': str(tmpdir.join('imports.dot')),
+                         'xref-file': str(tmpdir.join('imports.xref')),
+                         'hiddenimports': [],
+                         'specnm': 'issue_5131_script'})
+    # Speedup: avoid analyzing base_library.zip
+    monkeypatch.setattr(analysis, 'PY3_BASE_MODULES', [])
+
+    orig_getImports = bindepend.getImports
+    monkeypatch.setattr(bindepend, "getImports", getImports)
+
+    pkg = (tmpdir / 'mypkg').mkdir()
+    init = pkg / ('__init__' + EXTENSION_SUFFIXES[0])
+    init.write_binary(b'\0\0\0\0\0\0\0\0\0\0\0\0' * 20)
+    script = tmpdir.join('script.py')
+    script.write('import mypkg')
+    a = Analysis([str(script)],
+                 excludes=['encodings', 'pydoc', 'xml', 'distutils'])
+    PYZ(a.pure, a.zipped_data)
diff --git a/tests/unit/test_building_utils.py b/tests/unit/test_building_utils.py
--- a/tests/unit/test_building_utils.py
+++ b/tests/unit/test_building_utils.py
@@ -12,6 +12,7 @@
 
 import pytest
 import os
+from importlib.machinery import EXTENSION_SUFFIXES
 
 from PyInstaller.building import utils
 
@@ -78,3 +79,17 @@ def _(path): return os.path.join(*path.split('/'))
 
     res = utils.format_binaries_and_datas(datas, str(tmpdir))
     assert res == expected
+
+
+def test_add_to_suffix__extension():
+    toc = [
+        ('mypkg',
+         'lib38/site-packages/mypkg' + EXTENSION_SUFFIXES[0],
+         'EXTENSION'),
+    ]
+    toc = utils.add_suffix_to_extensions(toc)
+    assert toc == [
+        ('mypkg' + EXTENSION_SUFFIXES[0],
+         'lib38/site-packages/mypkg' + EXTENSION_SUFFIXES[0],
+         'EXTENSION'),
+    ]
diff --git a/tests/unit/test_modulegraph/test_modulegraph.py b/tests/unit/test_modulegraph/test_modulegraph.py
--- a/tests/unit/test_modulegraph/test_modulegraph.py
+++ b/tests/unit/test_modulegraph/test_modulegraph.py
@@ -12,6 +12,9 @@
 from lxml import etree
 import pickle
 
+from importlib._bootstrap_external import SourceFileLoader, ExtensionFileLoader
+from zipimport import zipimporter
+
 try:
     bytes
 except NameError:
@@ -188,25 +191,21 @@ def test_find_module(self):
             modgraph = modulegraph.ModuleGraph()
             info = modgraph._find_module('mymodule', path=[path] + sys.path)
 
-            fp = info[0]
-            filename = info[1]
-            description = info[2]
-
-            self.assertTrue(hasattr(fp, 'read'))
+            filename, loader = info
 
             if path.endswith('.zip') or path.endswith('.egg'):
                 # Zip importers may precompile
                 if filename.endswith('.py'):
                     self.assertEqual(filename, os.path.join(path, 'mymodule.py'))
-                    self.assertEqual(description, ('.py', READ_MODE, imp.PY_SOURCE))
+                    self.assertIsInstance(loader, zipimporter)
 
                 else:
                     self.assertEqual(filename, os.path.join(path, 'mymodule.pyc'))
-                    self.assertEqual(description, ('.pyc', 'rb', imp.PY_COMPILED))
+                    self.assertIsInstance(loader, zipimporter)
 
             else:
                 self.assertEqual(filename, os.path.join(path, 'mymodule.py'))
-                self.assertEqual(description, ('.py', READ_MODE, imp.PY_SOURCE))
+                self.assertIsInstance(loader, SourceFileLoader)
 
             # Compiled plain module, no source
             if path.endswith('.zip') or path.endswith('.egg'):
@@ -215,15 +214,10 @@ def test_find_module(self):
             else:
                 info = modgraph._find_module('mymodule2', path=[path] + sys.path)
 
-                fp = info[0]
-                filename = info[1]
-                description = info[2]
+                filename, loader = info
 
-                self.assertTrue(hasattr(fp, 'read'))
                 self.assertEqual(filename, os.path.join(path, 'mymodule2.pyc'))
-                self.assertEqual(description, ('.pyc', 'rb', imp.PY_COMPILED))
 
-                fp.close()
 
             # Compiled plain module, with source
 #            info = modgraph._find_module('mymodule3', path=[path] + sys.path)
@@ -243,14 +237,12 @@ def test_find_module(self):
 
             # Package
             info = modgraph._find_module('mypkg', path=[path] + sys.path)
-            fp = info[0]
-            filename = info[1]
-            description = info[2]
 
-            self.assertEqual(fp, None)
+            filename, loader = info
+
             # FIXME: PyInstaller appends `__init__.py` to the pkg-directory
             #self.assertEqual(filename, os.path.join(path, 'mypkg'))
-            self.assertEqual(description, ('', '', imp.PKG_DIRECTORY))
+            self.assertTrue(loader.is_package("mypkg"))
 
             # Extension
             if path.endswith('.zip'):
@@ -276,9 +268,7 @@ def test_find_module(self):
                 pass
             else:
                 info = modgraph._find_module('myext', path=[path] + sys.path)
-                fp = info[0]
-                filename = info[1]
-                description = info[2]
+                filename, loader = info
 
                 if sys.platform == 'win32':
                     ext = '.pyd'
@@ -287,8 +277,7 @@ def test_find_module(self):
                     ext = '.so'
 
                 self.assertEqual(filename, os.path.join(path, 'myext' + ext))
-                self.assertEqual(description, (ext, 'rb', imp.C_EXTENSION))
-                self.assertEqual(fp, None)
+                self.assertIsInstance(loader, ExtensionFileLoader)
 
     def test_moduleInfoForPath(self):
         self.assertEqual(modulegraph.moduleInfoForPath("/somewhere/else/file.txt"), None)
@@ -879,7 +868,7 @@ def _find_module(self, name, path, parent=None):
             graph = modulegraph.ModuleGraph()
             m = graph._find_module('sys', None)
             self.assertEqual(record, [])
-            self.assertEqual(m, (None, None, ("", "", imp.C_BUILTIN)))
+            self.assertEqual(m, (None, modulegraph.BUILTIN_MODULE))
 
             xml = graph.import_hook("xml")[0]
             self.assertEqual(xml.identifier, 'xml')
@@ -892,19 +881,15 @@ def _find_module(self, name, path, parent=None):
                 ('shutil', graph.path),
             ])
             self.assertTrue(isinstance(m, tuple))
-            self.assertEqual(len(m), 3)
-            self.assertTrue(hasattr(m[0], 'read'))
-            self.assertIsInstance(m[0].read(), str)
+            self.assertEqual(len(m), 2)
             srcfn = shutil.__file__
             if srcfn.endswith('.pyc'):
                 srcfn = srcfn[:-1]
-            self.assertEqual(os.path.realpath(m[1]), os.path.realpath(srcfn))
-            self.assertEqual(m[2], ('.py', READ_MODE, imp.PY_SOURCE))
-            m[0].close()
+            self.assertEqual(os.path.realpath(m[0]), os.path.realpath(srcfn))
+            self.assertIsInstance(m[1], SourceFileLoader)
 
             m2 = graph._find_module('shutil', None)
             self.assertEqual(m[1:], m2[1:])
-            m2[0].close()
 
 
             record[:] = []
@@ -916,7 +901,6 @@ def _find_module(self, name, path, parent=None):
             self.assertEqual(record, [
                 ('sax', xml.packagepath),
             ])
-            if m[0] is not None: m[0].close()
 
         finally:
             pass
diff --git a/tests/unit/test_modulegraph_more.py b/tests/unit/test_modulegraph_more.py
--- a/tests/unit/test_modulegraph_more.py
+++ b/tests/unit/test_modulegraph_more.py
@@ -16,6 +16,7 @@
 import py_compile
 import textwrap
 import zipfile
+from importlib.machinery import EXTENSION_SUFFIXES
 
 import pytest
 
@@ -69,6 +70,55 @@ def test_package(tmpdir):
     assert node.packagepath == [pysrc.dirname]
 
 
+#-- Extension modules
+
+@pytest.mark.parametrize(
+    "num, modname, expected_nodetype", (
+        # package's __init__ module is an extension
+        (1, "myextpkg", modulegraph.ExtensionPackage),
+        # __init__.py beside the __init__ module being an extension
+        (2, "myextpkg", modulegraph.ExtensionPackage),
+        # Importing a module beside
+        (3, "myextpkg.other", modulegraph.Extension),
+        # sub-package's __init__ module is an extension
+        (4, "myextpkg.subpkg", modulegraph.ExtensionPackage),
+        # importing a module beside, but from a sub-package
+        (5, "myextpkg.subpkg.other", modulegraph.Extension),
+    ))
+def test_package_init_is_extension(tmpdir, num, modname, expected_nodetype):
+    # Regression: Recursion to deep
+
+    def wt(*args):
+        f = tmpdir.join(*args)
+        f.write_text('###', encoding="ascii")
+        return f
+
+    def create_package_files(test_case):
+        (tmpdir / 'myextpkg' / 'subpkg').ensure(dir=True)
+        m = wt('myextpkg', '__init__' + EXTENSION_SUFFIXES[0])
+        if test_case == 1: return m  # noqa: E701
+        wt('myextpkg', '__init__.py')
+        if test_case == 2: return m  # noqa: E701 return extension module anway
+        m = wt('myextpkg', 'other.py')
+        m = wt('myextpkg', 'other' + EXTENSION_SUFFIXES[0])
+        if test_case == 3: return m  # noqa: E701
+        m = wt('myextpkg', 'subpkg', '__init__.py')
+        m = wt('myextpkg', 'subpkg', '__init__' + EXTENSION_SUFFIXES[0])
+        if test_case == 4: return m  # noqa: E701
+        m = wt('myextpkg', 'subpkg', 'other.py')
+        m = wt('myextpkg', 'subpkg', 'other' + EXTENSION_SUFFIXES[0])
+        return m
+
+    module_file = create_package_files(num)
+    node = _import_and_get_node(tmpdir, modname)
+    assert node.__class__ is expected_nodetype
+    if expected_nodetype is modulegraph.ExtensionPackage:
+        assert node.packagepath == [module_file.dirname]
+    else:
+        assert node.packagepath is None  # not a package
+    assert node.filename == str(module_file)
+
+
 #-- Basic tests - these seem to be missing in the original modulegraph
 #-- test-suite
 
@@ -196,11 +246,11 @@ def test_import_order_1(tmpdir):
     # Ensure modulegraph processes modules in the same order as Python does.
 
     class MyModuleGraph(modulegraph.ModuleGraph):
-        def _load_module(self, fqname, fp, pathname, info):
+        def _load_module(self, fqname, pathname, loader):
             if not record or record[-1] != fqname:
                 record.append(fqname) # record non-consecutive entries
-            return super(MyModuleGraph, self)._load_module(fqname, fp,
-                                                           pathname, info)
+            return super(MyModuleGraph, self)._load_module(fqname,
+                                                           pathname, loader)
 
     record = []
 
@@ -239,11 +289,11 @@ def test_import_order_2(tmpdir):
     # Ensure modulegraph processes modules in the same order as Python does.
 
     class MyModuleGraph(modulegraph.ModuleGraph):
-        def _load_module(self, fqname, fp, pathname, info):
+        def _load_module(self, fqname, pathname, loader):
             if not record or record[-1] != fqname:
                 record.append(fqname) # record non-consecutive entries
-            return super(MyModuleGraph, self)._load_module(fqname, fp,
-                                                           pathname, info)
+            return super(MyModuleGraph, self)._load_module(fqname,
+                                                           pathname, loader)
 
     record = []