Repository: pyinstaller/pyinstaller

Issue:
indexerror in depend/bytecode.py
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the mailing list, see
	  <http://www.pyinstaller.org/support.html#mailing-list>
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue
Having Error while building .exe and it seems an internal issue.
Note: Rebuilded the bootloader with VS latest
Note: Again tried with clean install from direct ....develop.zip
[Edit-1:] Re run with `--dbug all` arg
### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.0.dev0```
* Version of Python: 3.7
* Platform: Windows 10 64bit (english)
* How you installed Python: python.org/downloads
* Did you also try this on another platform? Does it work there?
prev dev version worked

* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [x] Package your program in **--onedir mode**
  * [x] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

```
# seems like an import issue. so specifying all imports
# required pypiwin32 comtypes pywin32-ctypes pafy youtube_dl cffi

from urllib import request, parse
from urllib.error import URLError, HTTPError
import pafy, random

from os import getcwd, remove, makedirs
from time import sleep
from os.path import exists
import re
from platform import system as os_name
os_name = os_name()
if os_name == "Windows":
    import console_mod
    console_mod.set_width(16)
    console_mod.enable_color()
"""
import sys
from ctypes import POINTER, WinDLL, Structure, sizeof, byref
from ctypes.wintypes import BOOL, SHORT, WCHAR, UINT, ULONG, DWORD, HANDLE
import ctypes
"""

from os import system as os_system

from platform import system as os_name
os_name = os_name()
import random
from sys	import getfilesystemencoding


if os_name == 'Windows':
	#from .windows import AudioClip as _PlatformSpecificAudioClip
	from ctypes import windll, c_buffer
	class PlaysoundException(Exception):
		pass

	# TODO: detect errors in all mci calls
	class _PlatformSpecificAudioClip(object):
		def directsend(self, command):
			buf = c_buffer(255)
			command = ''.join(command).encode(getfilesystemencoding())
			errorCode = int(windll.winmm.mciSendStringA(command, buf, 254, 0))
			if errorCode:
				errorBuffer = c_buffer(255)
				windll.winmm.mciGetErrorStringA(errorCode, errorBuffer, 254)
				exceptionMessage = ('\n	Error ' + str(errorCode) + ' for command:'
									'\n		' + command.decode() +
									'\n	' + errorBuffer.value.decode())
				raise PlaysoundException(exceptionMessage)
			return buf.value


```

### Stacktrace / full error message


```
py -3.7-64 -O -m PyInstaller "yt_plugin.py" -n "Youtube Player CLI" --version-file vtesty.py -i "ic.ico" --noupx --onedir --debug all
171 INFO: PyInstaller: 5.0.dev0
173 INFO: Python: 3.7.9
174 INFO: Platform: Windows-10-10.0.22000-SP0176 INFO: wrote E:\Ratul Codes\C\Python\Project_Yui\Yt_plugin_Standalone\Youtube Player CLI.spec
364 INFO: UPX is available.
369 INFO: Extending PYTHONPATH with paths
['E:\\Ratul Codes\\C\\Python\\Project_Yui\\Yt_plugin_Standalone',
 'E:\\Ratul Codes\\C\\Python\\Project_Yui\\Yt_plugin_Standalone']
425 INFO: checking Analysis
500 INFO: Building because Analysis-00.toc is bad
502 INFO: Initializing module dependency graph...
510 INFO: Caching module graph hooks...
551 INFO: Analyzing base_library.zip ...
5155 INFO: Processing pre-find module path hook distutils from 'D:\\PYTHON\\python37\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-distutils.py'.
5179 INFO: distutils: retargeting to non-venv dir 'D:\\PYTHON\\python37\\lib'
7893 INFO: Caching module dependency graph...
8352 INFO: running Analysis Analysis-00.toc
8416 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by D:\PYTHON\python37\python.exe
9736 INFO: Analyzing E:\Ratul Codes\C\Python\Project_Yui\Yt_plugin_Standalone\yt_plugin.py
21007 INFO: Processing module hooks...
21010 INFO: Loading module hook 'hook-difflib.py' from 'D:\\PYTHON\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
21025 INFO: Loading module hook 'hook-distutils.py' from 'D:\\PYTHON\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
21028 INFO: Loading module hook 'hook-encodings.py' from 'D:\\PYTHON\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
21494 INFO: Loading module hook 'hook-heapq.py' from 'D:\\PYTHON\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
21504 INFO: Loading module hook 'hook-pickle.py' from 'D:\\PYTHON\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
21513 INFO: Loading module hook 'hook-sysconfig.py' from 'D:\\PYTHON\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
21516 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from 'D:\\PYTHON\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
21519 INFO: Loading module hook 'hook-xml.py' from 'D:\\PYTHON\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
21660 INFO: Looking for ctypes DLLs
Traceback (most recent call last):
  File "D:\PYTHON\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "D:\PYTHON\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\__main__.py", line 136, in <module>  
    run()
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\__main__.py", line 126, in run       
    run_build(pyi_config, spec_file, **vars(args))
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\__main__.py", line 65, in run_build  
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\building\build_main.py", line 758, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\building\build_main.py", line 705, in build
    exec(code, spec_namespace)
  File "E:\Ratul Codes\C\Python\Project_Yui\Yt_plugin_Standalone\Youtube Player CLI.spec", line 19, in <module>
    noarchive=True)
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\building\build_main.py", line 254, in __init__
    self.__postinit__()
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\building\datastruct.py", line 159, in __postinit__
    self.assemble()
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\building\build_main.py", line 453, in assemble
    ctypes_binaries = scan_code_for_ctypes(co)
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\depend\utils.py", line 122, in scan_code_for_ctypes
    binaries = __recursively_scan_code_objects_for_ctypes(co)
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\depend\utils.py", line 198, in __recursively_scan_code_objects_for_ctypes
    binaries.extend(i)
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\depend\utils.py", line 223, in _scan_code_for_ctypes_getattr
    name = bytecode.load(name, code)
  File "D:\PYTHON\python37\lib\site-packages\PyInstaller\depend\bytecode.py", line 128, in load
    return code.co_varnames[index]
IndexError: tuple index out of range
```

Please also see <https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs>
for more about what would use to solve the issue.



Discussion:
ok now things are getting ugly.
Tried on both 3.7.9 (32 and 64bit)
both 5.0(dev) and 4.4 caused same error. 
but without any change, just using 4.3 compiles it successfully.
so i believe its a issue in the newer version. Please take a look. If you want i may send you the source code too.
Yeah that feature is new so you'll get in on pyinstaller >= 4.4 but not before.
is there any fix coming or i have make changes in my code??
I'm working on fixing it. But I first need to known which file it failed to scan to know why it failed to scan. Can you rebuild with `--log-level=DEBUG`? It should give some lines saying `11688 DEBUG: Scanning youtube_dl.compat for shared libraries or dlls` before it dies.
Whichever file it says last, please find and send me that file.
`youtube_dl.YoutubeDL` is the offending module.
This is actually simpler than it looks. The regex matched a block starting on an odd byte. By which I mean, it's supposed to read bytecode in pairs of bytes so every match's start and end indices should be multiples of two, but it has found a block of bytecode which *looks* like a LOAD operation if you start reading it halfway through a bytecode pair. So it all really needs is a `if match.start() % 2: continue` somewhere.
Hmm... and after unaligned matches (those starting on odd bytes) are filtered out, we don't have to worry about any other potential false positive matches of `_ctypes_getattr_regex` (or any other bytecode regex, really), because bytecode does not permit longer arbitrary sequences (every other byte must be a valid opcode, since instructions are always 2 bytes long). Is that correct or is there some other catch?
Correct. No instruction ever starts on an odd byte so every even byte is an opcode and every odd byte is an argument. Even if an argument needs more than one byte to store (e.g. an integer >= 256), it gets split into a sequence of opcode/argument pairs where the arguments concatenated form the long argument.

Resolving patch:
diff --git a/PyInstaller/building/build_main.py b/PyInstaller/building/build_main.py
--- a/PyInstaller/building/build_main.py
+++ b/PyInstaller/building/build_main.py
@@ -450,8 +450,12 @@ def assemble(self):
         for name, co in ctypes_code_objs.items():
             # Get dlls that might be needed by ctypes.
             logger.debug('Scanning %s for shared libraries or dlls', name)
-            ctypes_binaries = scan_code_for_ctypes(co)
-            self.binaries.extend(set(ctypes_binaries))
+            try:
+                ctypes_binaries = scan_code_for_ctypes(co)
+                self.binaries.extend(set(ctypes_binaries))
+            except Exception as ex:
+                raise RuntimeError(f"Failed to scan the module '{name}'. "
+                                   f"This is a bug. Please report it.") from ex
 
         self.datas.extend(
             (dest, source, "DATA") for (dest, source) in
diff --git a/PyInstaller/depend/bytecode.py b/PyInstaller/depend/bytecode.py
--- a/PyInstaller/depend/bytecode.py
+++ b/PyInstaller/depend/bytecode.py
@@ -25,6 +25,7 @@
 import dis
 import re
 from types import CodeType
+from typing import Pattern
 
 
 def _instruction_to_regex(x: str):
@@ -64,6 +65,32 @@ def bytecode_regex(pattern: bytes, flags=re.VERBOSE | re.DOTALL):
     return re.compile(pattern, flags=flags)
 
 
+def finditer(pattern: Pattern, string):
+    """Call ``pattern.finditer(string)`` but remove any matches beginning on an
+    odd byte. i.e. match.start() is not a multiple of 2.
+
+    This should be used to avoid false positive matches where a bytecode pair's
+    argument is mistaken for an opcode.
+    """
+    matches = pattern.finditer(string)
+    while True:
+        for match in matches:
+            if match.start() % 2 == 0:
+                # All is good. This match starts on an OPCODE.
+                yield match
+            else:
+                # This match has started on an odd byte meaning that it's a
+                # false positive and should be skipped. There is a very slim
+                # chance that a genuine match overlaps this one and, because
+                # re.finditer() doesn't allow overlapping matches, it would be
+                # lost. To avoid that, restart the regex scan starting at the
+                # next even byte.
+                matches = pattern.finditer(string, match.start() + 1)
+                break
+        else:
+            break
+
+
 # language=PythonVerboseRegExp
 _call_function_bytecode = bytecode_regex(rb"""
     # Matches `global_function('some', 'constant', 'arguments')`.
@@ -150,7 +177,7 @@ def function_calls(code: CodeType) -> list:
     match: re.Match
     out = []
 
-    for match in _call_function_bytecode.finditer(code.co_code):
+    for match in finditer(_call_function_bytecode, code.co_code):
         function_root, methods, args, arg_count = match.groups()
 
         # For foo():
diff --git a/PyInstaller/depend/utils.py b/PyInstaller/depend/utils.py
--- a/PyInstaller/depend/utils.py
+++ b/PyInstaller/depend/utils.py
@@ -219,7 +219,8 @@ def _scan_code_for_ctypes_getattr(code: CodeType):
 
     key_names = ("cdll", "oledll", "pydll", "windll")
 
-    for (name, attrs) in _ctypes_getattr_regex.findall(code.co_code):
+    for match in bytecode.finditer(_ctypes_getattr_regex, code.co_code):
+        name, attrs = match.groups()
         name = bytecode.load(name, code)
         attrs = bytecode.loads(attrs, code)
 


Test patch:
diff --git a/tests/unit/test_bytecode.py b/tests/unit/test_bytecode.py
--- a/tests/unit/test_bytecode.py
+++ b/tests/unit/test_bytecode.py
@@ -3,6 +3,7 @@
 Tests for PyInstaller.depend.bytecode
 """
 
+import re
 from types import CodeType
 from textwrap import dedent, indent
 import operator
@@ -12,6 +13,7 @@
     function_calls,
     recursive_function_calls,
     any_alias,
+    finditer,
 )
 
 
@@ -177,3 +179,26 @@ def test_local_functions():
 
 def test_any_alias():
     assert tuple(any_alias("foo.bar.pop")) == ("foo.bar.pop", "bar.pop", "pop")
+
+
+def test_finditer():
+    """Test that bytecode.finditer() yields matches only that start on an even
+     byte (``match.start() % 2 == 0``).
+
+    There are 3 permutations here when considering a match:
+    - A match starts on an even byte:
+        That's good! Include that sequence.
+    - A single character match starts on an odd byte:
+        Ignore it. It's a false positive.
+    - A multi-character match starts on an odd byte:
+        This match will be a false positive but there may be a genuine match
+        shortly afterwards (in the case of the test below - it'll be the next
+        character) which overlaps with this one so we must override regex's
+        behaviour of ignoring overlapping matches to prevent these from getting
+        lost.
+
+    """
+    matches = list(finditer(re.compile(r"\d+"),
+                            "0123" " 456" "7 89" "0 12" " 3 4"))
+    aligned = [i.group() for i in matches]
+    assert aligned == ["0123", "567", "890", "12"]