Repository: pyinstaller/pyinstaller

Issue:
inspect.getmodule fails for pyinstaller executable
##  `inspect.getmodule()` will not return module for pyinstaller executables.

### I have added jsonargparse to my program and they are using the following code:
`caller = inspect.getmodule(inspect.stack()[1][0]).__package__`
https://github.com/omni-us/jsonargparse/blob/9ae29b940285a4fa1ff7cba55099ab884874127a/jsonargparse/core.py#L241

`inspect.getmodule()` will return `None` for programs packaged with pyinstaller only. The program quits with the error:
```
Traceback (most recent call last):
  File "main.py", line 28, in <module>
    main()
  File "main.py", line 24, in main
    cfg = parser.parse_args()
  File "jsonargparse\core.py", line 379, in parse_args
  File "jsonargparse\core.py", line 242, in parse_known_args
AttributeError: 'NoneType' object has no attribute '__package__'
[3456] Failed to execute script main
```

* Output of `pyinstaller --version`: ```4.3```
* Version of Python: 3.8.6
* Platform: Windows 10 german
* How you installed Python: e.g. python.org/downloads
* Did you also try this on another platform? Does it work there?
No other platform available

* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```
No changes with latest version.

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [x] Package your program in **--onedir mode**
  * [x] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.
No matter if -F or not


### A minimal example program which shows the error
The minimal code removed even the jsonargparse library and added the command which should return the module but does only return None.
```
import inspect

def main():
    print("stack[1][0]:")
    print(inspect.stack()[1][0])
    print("getmodule(stack[1][0]):")
    module = inspect.getmodule(inspect.stack()[1][0])
    print(module)
    assert(module is not None)


if __name__ == '__main__':
    main()
```

### Stacktrace / full error message

```
stack[1][0]:
<frame at 0x00000220014812C0, file 'main.py', line 13, code <module>>
getmodule(stack[1][0]):
None <<< Should not be None (e.g. running directly: <module '__main__' from 'C:/git/pyinstaller-jsonargparse/main.py'>

Traceback (most recent call last):
  File "main.py", line 13, in <module>
    main()
  File "main.py", line 9, in main
    assert(module is not None)
AssertionError
```

Log Output of direct python call:
```
C:\git\pyinstaller-jsonargparse\venv\Scripts\python.exe C:/git/pyinstaller-jsonargparse/main.py
stack[1][0]:
<frame at 0x00000237AD899840, file 'C:/git/pyinstaller-jsonargparse/main.py', line 13, code <module>>
getmodule(stack[1][0]):
<module '__main__' from 'C:/git/pyinstaller-jsonargparse/main.py'>

Process finished with exit code 0
```

`inspect.getmodule()` has several module detection mechanisms but it seems no one is compatible with pyinstaller.
Currently it returns because the following expression returns true:     if not hasattr(object, '__name__')`
```
def getmodule(object, _filename=None):
    """Return the module an object was defined in, or None if not found."""
    if ismodule(object):
        return object
    if hasattr(object, '__module__'):
        return sys.modules.get(object.__module__)
    # Try the filename to modulename cache
    if _filename is not None and _filename in modulesbyfile:
        return sys.modules.get(modulesbyfile[_filename])
    # Try the cache again with the absolute file name
    try:
        file = getabsfile(object, _filename)
    except TypeError:
        return None
    if file in modulesbyfile:
        return sys.modules.get(modulesbyfile[file])
    # Update the filename to module name cache and check yet again
    # Copy sys.modules in order to cope with changes while iterating
    for modname, module in sys.modules.copy().items():
        if ismodule(module) and hasattr(module, '__file__'):
            f = module.__file__
            if f == _filesbymodname.get(modname, None):
                # Have already mapped this module, so skip it
                continue
            _filesbymodname[modname] = f
            f = getabsfile(module)
            # Always map to the name the module knows itself by
            modulesbyfile[f] = modulesbyfile[
                os.path.realpath(f)] = module.__name__
    if file in modulesbyfile:
        return sys.modules.get(modulesbyfile[file])
    # Check the main module
    main = sys.modules['__main__']
    if not hasattr(object, '__name__'):
        return None
    if hasattr(main, object.__name__):
        mainobject = getattr(main, object.__name__)
        if mainobject is object:
            return main
    # Check builtins
    builtin = sys.modules['builtins']
    if hasattr(builtin, object.__name__):
        builtinobject = getattr(builtin, object.__name__)
        if builtinobject is object:
            return builtin
```


Discussion:
This looks like a (slightly different) variant of #5897 and probably due to behavior of `__main__` in frozen application not fully conforming to python's behavior. 

Running the provided test program as `python program.py` returns `module` as `<module '__main__' from '[...]/program.py'>`, whereas running it as `cat program.py | python` returns `<module '__main__' (built-in)>`. Running as frozen application, however, returns `None`.
It turns out that the problem here is not `__main__` per-se, but rather the path handling in `inspect` when it tries to resolve the calling module from stack frame (in this case, `__main__`).

The stack-frame object seems to provide only base-name of the "source" file (e.g., `program.py`), and `getabsfile`/`getsourcefile` seem to be trying to find it in current working directory.

That's why the test snippet seems to work if called from within `dist/program` folder (because then resolved name `[fullpath/]dist/program/program.py` coincides with `__main__.__file__`). It does not work from any other location, as
`[cur_location]/program.py` does not match `__main__.__file__`. And notably, it fails if called from directory containing the original source `program.py`, because then the latter "shadows" the fictious `program.py` declared in `__main__.__file__`.

One way (or rather, the only one I can think of) to solve this is by rthook that overrides the `inspect.getsourcefile` to always return `__main__.__file__` if the given filename matches `basename(__main__.__file__)`. Although, this also begs the question how `module = inspect.getmodule(inspect.stack()[1][0])` behaves when called from any other module...

Thank you for the insights! Can you give me a hint how i do such a hook? Can i do that on my own or needs this be done within the project?

Resolving patch:
diff --git a/PyInstaller/hooks/rthooks/pyi_rth_inspect.py b/PyInstaller/hooks/rthooks/pyi_rth_inspect.py
new file mode 100644
--- /dev/null
+++ b/PyInstaller/hooks/rthooks/pyi_rth_inspect.py
@@ -0,0 +1,51 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2021, PyInstaller Development Team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: Apache-2.0
+#-----------------------------------------------------------------------------
+
+
+import sys
+import os
+import inspect
+
+
+_orig_inspect_getsourcefile = inspect.getsourcefile
+
+
+# Provide custom implementation of inspect.getsourcefile() for frozen
+# applications that properly resolves relative filenames obtained from
+# object (e.g., inspect stack-frames). See #5963.
+def _pyi_getsourcefile(object):
+    filename = inspect.getfile(object)
+    if not os.path.isabs(filename):
+        # Check if given filename matches the basename of __main__'s
+        # __file__
+        main_file = sys.modules['__main__'].__file__
+        if filename == os.path.basename(main_file):
+            return main_file
+
+        # If filename ends with .py suffix and does not correspond to
+        # frozen entry-point script, convert it to corresponding .pyc
+        # in sys._MEIPASS
+        if filename.endswith('.py'):
+            filename = os.path.normpath(
+                os.path.join(sys._MEIPASS, filename + 'c'))
+            # Ensure the relative path did not try to jump out of
+            # sys._MEIPASS, just in case...
+            if filename.startswith(sys._MEIPASS):
+                return filename
+    elif filename.startswith(sys._MEIPASS) and filename.endswith('.pyc'):
+        # If filename is already PyInstaller-compatible, prevent any
+        # further processing (i.e., with original implementation)
+        return filename
+    # Use original implementation as a fallback
+    return _orig_inspect_getsourcefile(object)
+
+
+inspect.getsourcefile = _pyi_getsourcefile


Test patch:
diff --git a/tests/functional/modules/pyi_inspect_getmodule_from_stackframes/helper_package/__init__.py b/tests/functional/modules/pyi_inspect_getmodule_from_stackframes/helper_package/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_inspect_getmodule_from_stackframes/helper_package/__init__.py
@@ -0,0 +1,6 @@
+from . import helper_module
+
+
+def test_call_chain():
+    # Add another level of indirection...
+    return helper_module.test_call_chain()
diff --git a/tests/functional/modules/pyi_inspect_getmodule_from_stackframes/helper_package/helper_module.py b/tests/functional/modules/pyi_inspect_getmodule_from_stackframes/helper_package/helper_module.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_inspect_getmodule_from_stackframes/helper_package/helper_module.py
@@ -0,0 +1,11 @@
+import inspect
+
+
+def test_call_chain():
+    modules = []
+    for idx, frame_info in enumerate(inspect.stack()):
+        #print(frame_info)
+        module = inspect.getmodule(frame_info[0])
+        assert module, f"Failed to get module for frame #{idx}: {frame_info}"
+        modules.append(module)
+    return modules
diff --git a/tests/functional/test_misc.py b/tests/functional/test_misc.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/test_misc.py
@@ -0,0 +1,55 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2021, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License (version 2
+# or later) with exception for distributing the bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
+#-----------------------------------------------------------------------------
+
+
+import os
+
+
+# Directory with testing modules used in some tests.
+_MODULES_DIR = os.path.join(
+    os.path.dirname(os.path.abspath(__file__)),
+    'modules'
+)
+
+
+# Test inspect.getmodule() on stack-frames obtained by inspect.stack().
+# Reproduces the issue reported by #5963 while expanding the test to
+# cover a package and its submodule in addition to the __main__ module.
+def test_inspect_getmodule_from_stackframes(pyi_builder):
+    pathex = os.path.join(_MODULES_DIR,
+                          'pyi_inspect_getmodule_from_stackframes')
+    # NOTE: run_from_path MUST be True, otherwise cwd + rel_path coincides
+    # with sys._MEIPASS + rel_path and masks the path resolving issue
+    # in onedir builds.
+    pyi_builder.test_source(
+        """
+        import helper_package
+
+        # helper_package.test_call_chain() calls eponymous function in
+        # helper_package.helper_module, which in turn uses inspect.stack()
+        # and inspect.getmodule() to obtain list of modules involved in
+        # the chain call.
+        modules = helper_package.test_call_chain()
+
+        # Expected call chain
+        expected_module_names = [
+            'helper_package.helper_module',
+            'helper_package',
+            '__main__'
+        ]
+
+        # All modules must have been resolved
+        assert not any(module is None for module in modules)
+
+        # Verify module names
+        module_names = [module.__name__ for module in modules]
+        assert module_names == expected_module_names
+        """, pyi_args=['--paths', pathex], run_from_path=True)