Repository: pyinstaller/pyinstaller-hooks-contrib (language: python)

Issue:
yapf FileNotFoundError
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the discussions forum.
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue
yapf is used in the groundingDINO project.
Using pyinstaller to build a onedir executable.
yapf is not included in the onedir/_internal packages, and so module not found error is encountered on running the frozen application.

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```6.10```
* Version of Python: <!-- e.g. 3.11 --> 3.9.13
* Platform: <!-- e.g GNU/Linux (distribution), Windows (language settings), OS X, FreeBSD --> Windows 11
* How you installed Python: <!-- e.g. python.org/downloads, conda, brew, pyenv, apt, Windows store --> python.org/downloads
* Did you also try this on another platform? Does it work there? Have not tried on another platform

I have already tried adding yapf to hiddenimports.

  * [x] start with clean installation: TRIED
  * [ ] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it: TRIED
  * [x] Package your program in **--onedir mode**: TRIED
  * [ ] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.: TRIED (that's how discovered the error msg)

On manually copy-pasting the yapf module from my python installation into the onedir/_internal directory, the application works as expected.
So this seems a problem of yapf not being recognised and exported properly.

I'm using a .spec file to construct the executable.
Exe runs fine until the feature/piece of code that uses the yapf module is encountered.
### A minimal example program which shows the error

```
main.py:
import yapf

pyinstaller main.py --onedir
```
FileNotFoundError: No such file or directory: `path/to/dist/main/_internal/yapf_third_party/_ylib2to3/Grammar.txt`
### Reported earlier in pyinstaller/pyinstaller#4939 in 2020 but closed as stale


Discussion / hints:
> FileNotFoundError: No such file or directory: `path/to/dist/main/_internal/yapf_third_party/_ylib2to3/Grammar.txt`

Looks like `yapf` requires data file from `yapf_third_party` package. Try adding [`--collect-data yapf_third_party`](https://pyinstaller.org/en/stable/usage.html#cmdoption-collect-data) to your PyInstaller command.
> Reported earlier in https://github.com/pyinstaller/pyinstaller/issues/4939 in 2020 but closed as stale

Note that pyinstaller/pyinstaller#4939 has nothing to do with this issue. You have a `FileNotFoundError`, that issue is just messing up their Python environments.
> Looks like `yapf` requires data file from `yapf_third_party` package. Try adding [`--collect-data yapf_third_party`](https://pyinstaller.org/en/stable/usage.html#cmdoption-collect-data) to your PyInstaller command.

Thanks! That did the trick!

Out of curiosity, when I inspect the _internal directory of the dist, it still doesn't have the yapf module dir where it has others like onnx, torch etc.
Though the code that imports the yapf module works without issue.
How does this happen? Is it not necessary that modules included in the build show up as directories in the _internal dir?

This will help me have a better understanding of pyinstaller to diagnose future issues, as I've encountered similar issues before.

Thanks a bunch!

> Note that pyinstaller/pyinstaller#4939 has nothing to do with this issue. You have a `FileNotFoundError`, that issue is just messing up their Python environments.

I'm sorry, you're right.
When the FileNotFoundError came up I inspected the _internal dir of the dist and didn't find the yapf module dir there, whereas others like onnx, torch were present; and assumed this meant that the yapf module wasn't included properly with the build.

I guess my understanding of this is incorrect.

If you have more details about this process, could you please share that?

Thanks!
Python files don't appear on the file system. They're bytecompiled and put in a zip-like archive inside the exe file.
Oh the base-library.zip file? Or inside the exe in the onedir mode too?

And all the module-named directories like onnxruntime, torch that appear in the _internal directory are only there to keep the data files associated with those modules?
> Oh the base-library.zip file? Or inside the exe in the onedir mode too?

No, `base-library.zip` contains the stdlib modules that we need to initialize python interpreter and for bootstrap. The rest of pure-python modules are collected into PYZ archive, which is embedded in the executable (both in onefile and onedir mode). You can use `pyi-archive_viewer <path-to-executable>` to inspect the embedded PKG archive, and the PYZ archive that is embedded within it.

> And all the module-named directories like onnxruntime, torch that appear in the _internal directory are only there to keep the data files associated with those modules?

Yes. Data files and binaries - bundled shared libraries and binary extension modules. Although for some packages, like `torch`, we also need to collect source .py files, so you will see those in the `torch` directory as well...

Sorry for returning late to this.

Very helpful details, thanks a lot! I'll check out the archive viewer as well, sounds quite useful.

> Although for some packages, like torch, we also need to collect source .py files, so you will see those in the torch directory as well...

Is this because of aten and how the python API wraps over it, or because some configs are stored in .py files?
If my package is having an issue being bundled by pyinstaller, what could be a signal that I need to do something like this (like you do for torch)?
Torch takes plain source code and feeds it to its special JIT compiler. Occasionally, other libraries try to use source code for things they shouldn't. They will almost always be using `inspect.getsource()` which raises a fairly self explanatory `OSError: could not get source code` error.

Fix patch:
diff --git a/_pyinstaller_hooks_contrib/stdhooks/hook-yapf_third_party.py b/_pyinstaller_hooks_contrib/stdhooks/hook-yapf_third_party.py
new file mode 100644
index 00000000..d138c53b
--- /dev/null
+++ b/_pyinstaller_hooks_contrib/stdhooks/hook-yapf_third_party.py
@@ -0,0 +1,15 @@
+# ------------------------------------------------------------------
+# Copyright (c) 2024 PyInstaller Development Team.
+#
+# This file is distributed under the terms of the GNU General Public
+# License (version 2.0 or later).
+#
+# The full license is available in LICENSE, distributed with
+# this software.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+# ------------------------------------------------------------------
+
+from PyInstaller.utils.hooks import collect_data_files
+
+datas = collect_data_files('yapf_third_party')
diff --git a/news/792.new.rst b/news/792.new.rst
new file mode 100644
index 00000000..112ddb38
--- /dev/null
+++ b/news/792.new.rst
@@ -0,0 +1,2 @@
+Add hook for ``yapf_third_party`` (part of ``yapf``) to collect its
+data files.


Test patch:
diff --git a/requirements-test-libraries.txt b/requirements-test-libraries.txt
index 8eff023b..341a9d5e 100644
--- a/requirements-test-libraries.txt
+++ b/requirements-test-libraries.txt
@@ -217,6 +217,7 @@ xarray==2024.7.0; python_version >= '3.9'
 tables==3.10.1; python_version >= '3.10'
 schwifty==2024.9.0  # pyup: != '2024.8.1'
 patool==2.4.0; python_version >= '3.10'
+yapf==0.40.2
 
 # ------------------- Platform (OS) specifics
 
diff --git a/tests/test_libraries.py b/tests/test_libraries.py
index 2c0896a3..f77c032e 100644
--- a/tests/test_libraries.py
+++ b/tests/test_libraries.py
@@ -2156,8 +2156,6 @@ def test_itk(pyi_builder):
 def test_slixmpp(pyi_builder):
     pyi_builder.test_source("""
         import slixmpp
-
-
         slixmpp.ClientXMPP('username', 'password')
     """)
 
@@ -2168,3 +2166,10 @@ def test_capstone(pyi_builder):
         import capstone
         capstone.__version__
     """)
+
+
+@importorskip('yapf')
+def test_yapf(pyi_builder):
+    pyi_builder.test_source("""
+        import yapf
+    """)