Repository: pyinstaller/pyinstaller

Issue:
AssertionError when using PyInstaller. Fixed when following line is commented in modulegraph.py: assert prev_insts[-2].opname == 'LOAD_CONST'
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the mailing list, see
	  <http://www.pyinstaller.org/support.html#mailing-list>
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

The following was posted on the google groups discussion and was suggested by @Legorooj to post it here:

PyInstaller version: 4.10
Version of Python: 3.6.8
Platform: Windows 10
Python installed from: [python.org](http://python.org/)
Didn't try on another platform.
Will try development version if necessary.

Clean installation.
Ran program from command window
Used --onedir mode

Error:
```\site-packages\PyInstaller\lib\modulegraph\modulegraph.py, line 2771, in _scan_bytecode()`
    assert prev_insts[-2].opname == 'LOAD_CONST'
AssertionError```

Command line used: `pyinstaller -F --clean --paths {paths --collect-all {name1} --collect-all {name2} script.py`

### Quick overview
The problem arises when using `.pyc` files for the modules in `name1` and `name2`.
The problem does not occur when I comment out [this](https://github.com/pyinstaller/pyinstaller/blob/3ea48fd875e786ecbb3c160209791aa11d55ae99/PyInstaller/lib/modulegraph/modulegraph.py#L2764) line in modulegraph.py.
When I run pyinstaller without line 2764 (linked above), everything runs to completion and the executable generated works as intended.
I've been trying to look through previous commits to try and understand the reason for the assert statement and why it is necessary. Could it perhaps no longer be necessary?

Root cause: `The prev_insts[-2].opname` variable is equal to `EXTENDED_ARG` thus causing the assert statement to fail.

There's been a few issues before related to `EXTENDED_ARG` (but not my exact problem) as well as commits.

- [Link](https://github.com/pyinstaller/pyinstaller/pull/5830)
- [Link](https://github.com/pyinstaller/pyinstaller/issues/6301#issuecomment-962519524)
- And this [link](https://github.com/pyinstaller/pyinstaller/commit/ed732c92eded85e6bca4232eedefee2906bbfa52) seems to be the commit for the first variation of the LOAD_CONST assert but it's not clear to me why it's there.

Based on what I found in PyInstaller's current [bytecode.py](https://github.com/pyinstaller/pyinstaller/blob/1e6a8d53f150cf24b574c32085f3745cbd2afaa6/PyInstaller/depend/bytecode.py#L13), it seems like it might need to be expanded on:

> The biggest clarity killer here is the ``EXTENDED_ARG`` opcode which can appear almost anywhere and therefore needs to be tiptoed around at every step. If this code needs to expand significantly, I would recommend an upgrade to a regex-based grammar parsing library such as Reparse. This way, little steps like unpacking ``EXTENDED_ARGS`` can be defined once then simply referenced forming a nice hierarchy rather than copied everywhere its needed.


Could you please provide some information about why this type of opcode requires an exception to be thrown and why PyInstaller works when the assertion is removed? Or just confirm with me whether or not it is safe for me to continue on with the line removed? Or even just a way to avoid running into this exception?

[Reference](https://docs.python.org/3/library/dis.html?highlight=load_const#opcode-LOAD_CONST) to extended_arg from dis.py.

Thank you,

Eduardo


Discussion:
Ouch; looks like `modulegraph`'s bytecode scanning was never adapted to the bytecode that contains the `EXTENDED_ARG` opcodes. This goes both for our vendored version as well as [the upstream version](https://github.com/ronaldoussoren/modulegraph/blob/5ab33ae8a67b9927086916c6f29dd8b14554e3b6/modulegraph/modulegraph.py#L1768-L1770).

Since this is the first time we're seeing the problem, it is probably a very specific corner case. Namely, that codepath is likely triggered only for bytecode-only (.pyc) modules. And second, `EXTENDED_ARG` opcodes would appear before `LOAD_CONST` opcodes preceding the `IMPORT_NAME` only if an import was made after declaring > 256 names (e.g., variables).

<details><summary>A simple reproducer</summary>
<p>

Generate a module, called `ugly_code.py` using the following script:
```python
with open("ugly_code.py", "w") as fp:
    for i in range(300):
        print(f"dummy_var{i} = {i}", file=fp)
    print("import os", file=fp)
```
and create an entry-point script that imports the module:
```python
# program.py
import ugly_code
```
Then ensure that the module is byte-compiled (e.g., taking a copy from `__pycache__` after running the entry-point script once), and remove the source.

Trying to freeze the entry-point script should now trigger the reported `AssertionError`.

</p></details>

----

>Could you please provide some information about why this type of opcode requires an exception to be thrown

Because the code was written when `EXTENDED_ARG` did not exist yet, so it assumes that the sequence of opcodes involving an `import` is `LOAD_CONST`, `LOAD_CONST`, `LOAD_NAME`. If you look at the two lines following the asserts, you will see that it interprets the opcodes' arguments as indices into `co.co_consts` list (= list of constants' names) to resolve "level" and "from". Which is valid interpretation if those are `LOAD_CONST` opcodes, and not if they are not (the `EXTENDED_ARG` opcodes should be collected, and their arguments would need to be used to modify the argument of the `LOAD_CONST` opcode).

> and why PyInstaller works when the assertion is removed? 

Because you were lucky and the incorrect interpretation of the `EXTENDED_ARG`'s argument value did not cause an out-of-bounds index access in the `co.co_consts` list.

> Or just confirm with me whether or not it is safe for me to continue on with the line removed? Or even just a way to avoid running into this exception?

Looks like it is safe in your case (with that particular module). But I suppose it would be safer to just skip the problematic opcode
```python
if prev_insts[-2].opname != "LOAD_CONST":
   continue
if prev_insts[-1].opname != "LOAD_CONST":
   continue
```
Worst case, you will miss an import that you will need to handle manually via hiddenimports. But you'd be missing it either way, even if you allowed incorrect interpretation of `EXTENDED_ARG`s, so you're not really any worse off...

---

All in all, while this is a shortcoming of the current `modulegraph` implementation (both our vendored version, and the upstream one), I'm not sure we'll be addressing it at the moment. It is a very specific corner case, plus within a year's time, we'll need to replace much of the `modulegraph`'s code anyway, and @Legorooj is working on the replacement.
And FWIW, if you are using byte-compiled version of modules because PyInstaller is otherwise collecting their source .py files, this is happening because you are using `--collect-all`, which should be avoided (use `--collect-submodules`, `--collect-data`, `--collect-binaries` to collect whatever part of the package you actually need to explicitly collect). So if you can use source .py files instead of .pycs, you should use them, and avoid this issue altogether.
That's odd. I thought that [`dis.get_instructions()`](https://docs.python.org/3.7/library/dis.html#dis.get_instructions) automatically dealt with and flattened out the `EXTENDED_ARG` opcode. On looking at it again, it looks like it automatically adjusts the arg number for the instruction it's extending but doesn't finish the job by removing the `EXTENDED_ARG` instruction from the output.
```python
>>> import ugly_code
>>> code = ugly_code.__loader__.get_code("ugly_code")
>>> import dis
>>> print(*list(dis.get_instructions(code))[-8:], sep="\n")
Instruction(opname='LOAD_CONST', opcode=100, arg=300, argval=None, argrepr='None', offset=1380, starts_line=None, is_jump_target=False)
Instruction(opname='EXTENDED_ARG', opcode=144, arg=1, argval=1, argrepr='', offset=1382, starts_line=None, is_jump_target=False)
Instruction(opname='IMPORT_NAME', opcode=108, arg=300, argval='os', argrepr='os', offset=1384, starts_line=None, is_jump_target=False)
Instruction(opname='EXTENDED_ARG', opcode=144, arg=1, argval=1, argrepr='', offset=1386, starts_line=None, is_jump_target=False)
Instruction(opname='STORE_NAME', opcode=90, arg=300, argval='os', argrepr='os', offset=1388, starts_line=None, is_jump_target=False)
Instruction(opname='EXTENDED_ARG', opcode=144, arg=1, argval=1, argrepr='', offset=1390, starts_line=None, is_jump_target=False)
Instruction(opname='LOAD_CONST', opcode=100, arg=300, argval=None, argrepr='None', offset=1392, starts_line=None, is_jump_target=False)
Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=1394, starts_line=None, is_jump_target=False)
```
In which case, all I think we need to do it add some `instructions = (i for i in instructions if i.opcode != EXTENDED_ARG)` guards into [here](https://github.com/pyinstaller/pyinstaller/blob/3ea48fd875e786ecbb3c160209791aa11d55ae99/PyInstaller/lib/modulegraph/util.py#L139)?
> In which case, all I think we need to do it add some `instructions = (i for i in instructions if i.opcode != EXTENDED_ARG)` guards into [here](https://github.com/pyinstaller/pyinstaller/blob/3ea48fd875e786ecbb3c160209791aa11d55ae99/PyInstaller/lib/modulegraph/util.py#L139)?

Good call. If we can just filter `EXTENDED_ARG` out  and not have to worry about actually extending the argument values, it's reasonable to fix this now even if we end up replacing all that code in the (near?) future. 

Resolving patch:
diff --git a/PyInstaller/lib/modulegraph/util.py b/PyInstaller/lib/modulegraph/util.py
--- a/PyInstaller/lib/modulegraph/util.py
+++ b/PyInstaller/lib/modulegraph/util.py
@@ -136,7 +136,9 @@ def iterate_instructions(code_object):
     Yields `dis.Instruction`. After each code-block (`co_code`), `None` is
     yielded to mark the end of the block and to interrupt the steam.
     """
-    yield from get_instructions(code_object)
+    # The arg extension the EXTENDED_ARG opcode represents is automatically handled by get_instructions() but the
+    # instruction is left in. Get rid of it to make subsequent parsing easier/safer.
+    yield from (i for i in get_instructions(code_object) if i.opname != "EXTENDED_ARG")
 
     yield None
 


Test patch:
diff --git a/tests/unit/test_modulegraph/test_imports.py b/tests/unit/test_modulegraph/test_imports.py
--- a/tests/unit/test_modulegraph/test_imports.py
+++ b/tests/unit/test_modulegraph/test_imports.py
@@ -531,5 +531,16 @@ def assertIsInstance(self, value, types):
                 self.fail("%r is not an instance of %r"%(value, types))
 
 
+def test_extended_args_import():
+    source = "".join(f"dummy_var{i} = {i}\n" for i in range(300)) + "import os\n"
+    code = compile(source, "", "exec")
+    node = modulegraph.Node("dummy_module")
+    node._deferred_imports = []
+    node.code = code
+    graph = modulegraph.ModuleGraph()
+    graph._scan_bytecode(node, code, True)
+    assert node._deferred_imports[0][1][0] == "os"
+
+
 if __name__ == "__main__":
     unittest.main()