Repository: spyder-ide/spyder

Issue:
Python seems to be incorrectly compiled for macOS installer
## Problem Description

The macOS standalone application reports the following warning in the IPython console when the Python interpreter is set to internal (same as Spyder):
```
This version of python seems to be incorrectly compiled
(internal generated filenames are not absolute).
This may make the debugger miss breakpoints.
Related bug: http://bugs.python.org/issue1666807
```

### What steps reproduce the problem?

1. Launch macOS standalone Spyder application with Python interpreter set to internal


## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->
* Spyder version: 5.2.0.dev0 None
* Python version: 3.9.5 64-bit
* Qt version: 5.12.10
* PyQt5 version: 5.12.3
* Operating System: Darwin 20.6.0

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
# Mandatory:
applaunchservices >=0.1.7     :  0.2.1 (OK)
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  4.0.0 (OK)
cloudpickle >=0.5.0           :  2.0.0 (OK)
cookiecutter >=1.6.0          :  1.7.3 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.1.0 (OK)
IPython >=7.6.0               :  7.29.0 (OK)
jedi >=0.17.2;<0.19.0         :  0.18.0 (OK)
jellyfish >=0.7               :  0.8.9 (OK)
jsonschema >=3.2.0            :  4.2.1 (OK)
keyring >=17.0.0              :  23.2.1 (OK)
nbconvert >=4.0               :  6.3.0 (OK)
numpydoc >=0.6.0              :  1.1.0 (OK)
parso >=0.7.0;<0.9.0          :  0.8.2 (OK)
pexpect >=4.4.0               :  4.8.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.8.0 (OK)
pygments >=2.0                :  2.10.0 (OK)
pylint >=2.5.0                :  2.9.6 (OK)
pyls_spyder >=0.4.0           :  0.4.0 (OK)
pylsp >=1.2.2;<1.3.0          :  1.3.0.dev0 (OK)
pylsp_black >=1.0.0           :  None (OK)
qdarkstyle =3.0.2             :  3.0.2 (OK)
qstylizer >=0.1.10            :  0.2.1 (OK)
qtawesome >=1.0.2             :  1.1.0 (OK)
qtconsole >=5.2.0;<5.3.0      :  5.2.0 (OK)
qtpy >=1.5.0                  :  1.11.2 (OK)
rtree >=0.9.7                 :  0.9.7 (OK)
setuptools >=49.6.0           :  58.5.3 (OK)
sphinx >=0.6.6                :  4.3.0 (OK)
spyder_kernels >=2.1.1;<2.2.0 :  2.2.0.dev0 (OK)
textdistance >=4.2.0          :  4.2.2 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3             :  2.1.6 (OK)
zmq >=17                      :  22.3.0 (OK)

# Optional:
cython >=0.21                 :  None (OK)
matplotlib >=2.0.0            :  None (OK)
numpy >=1.7                   :  None (OK)
pandas >=1.1.1                :  None (OK)
scipy >=0.17.0                :  None (OK)
sympy >=0.7.3                 :  None (OK)
```



Discussion:
Just a quick note: I wonder if this could be a problem with the latest Py2app (in case we're using it).
The provided link, http://bugs.python.org/issue1666807, indicates an issue with absolute paths in pyc files that was fixed over 10 years ago (https://bugs.python.org/issue1180193). Nevertheless, the same symptoms manifest in the IPython console when using the internal Python interpreter.

```python
In [1]: import inspect

In [2]: inspect.getabsfile(inspect)
Out[2]: '/Users/rclary/Applications/spyder-dev/Spyder.app/Contents/Resources/lib/python39.zip/inspect.py'

In [3]:inspect.getabsfile(inspect.iscode)
Out[3]: '/Users/rclary/inspect.py'

In [4]: inspect.__file__
Out[4]: '/Users/rclary/Applications/spyder-dev/Spyder.app/Contents/Resources/lib/python39.zip/inspect.pyc'

In [5]: import jedi

In [6]: jedi.__file__
Out[6]: '/Users/rclary/Applications/spyder-dev/Spyder.app/Contents/Resources/lib/python3.9/jedi/__init__.py'

In [7]: inspect.getabsfile(jedi)
Out[7]: '/Users/rclary/Applications/spyder-dev/Spyder.app/Contents/Resources/lib/python3.9/jedi/__init__.py'

In [8]: inspect.getabsfile(jedi.get_system_environment)
Out[8]: '/Users/rclary/Applications/spyder-dev/Spyder.app/Contents/Resources/lib/python3.9/jedi/api/environment.py'
```

Above you can see that importing the module from the zip folder inside the application resource path (a pyc file) results in `inspect` using paths relative to the current working directory rather than relative to the module path. This is not the case when the module is loaded from a python source file (py file).

I don't know if this issue is malignant or benign. I'm not aware of any adverse behavior in our macOS standalone applications when using the internal interpreter for the IPython console; so it would seem, so far, that the only real issue is the warning message displayed on IPython console startup. This issue does not manifest when using an external interpreter.

This warning message does not appear in our 5.1.5 release, but does appear in the artifact for PR #16322. As of this investigation, this was the oldest merged PR for which artifacts still remained on the server (GitHub only keeps them for 30 days).
> This warning message does not appear in our 5.1.5 release

That's why I thought that maybe this could be related to a newer version of Py2app (or perhaps one of its dependencies).

> but does appear in the artifact for PR #16322

I really doubt that PR introduced this problem.

----

Two additional questions:

* If everything seems to be working with the internal interpreter, perhaps we should patch Python to avoid showing that warning?
* Do you have time to do that (or keep investigating) this week? Our plan is to release 5.2.0 next Monday.
> > This warning message does not appear in our 5.1.5 release
> 
> That's why I thought that maybe this could be related to a newer version of Py2app (or perhaps one of its dependencies).
> 
> > but does appear in the artifact for PR #16322
> 
> I really doubt that PR introduced this problem.

I agree that our source code did not introduce this problem; that was merely the first artifact available on the server for which the issue manifested. I think it was revealed with IPython 7.28.0.

We are still using `py2app==0.22`, so this is not the issue. There are several updates to py2app but it has introduced a Qt plugin issue that prohibits us from updating (ronaldoussoren/py2app#380). In my local patches for py2app's latest commits, this warning message in the IPython console does not appear, although the symptoms exhibited by `inpsect` shown above still manifest. This local build uses `ipython==7.29.0` and `jupyter-core==4.9.1`.

> Two additional questions:
> 
>     * If everything seems to be working with the internal interpreter, perhaps we should patch Python to avoid showing that warning?
> 
>     * Do you have time to do that (or keep investigating) this week? Our plan is to release 5.2.0 next Monday.

It appears that the warning message began manifesting upon `ipython>=7.28.0`, and limiting it to 7.27.0 eliminates the warning. I recommend the following:
* For now, limit `ipython<7.28.0` for just the macOS installer.
* This constraint will be removed upon updating `py2app>=0.27`
* It still remains to be determined whether this is merely covering up the issue and whether this issue is malignant or benign. I don't think we can know until a concrete behavioral issue is reported by a Spyder user.




Ok, thanks a lot for your help with this one @mrclary!
Per conversations in #16795 and #14025, reopening this issue.
# Update
I've discovered the following:
* The warning message originates in the module `debugpy._vendored.pydevd.pydevd_file_utils.py` at the module level.
* The message is a `pydev_log.critical` logger message conditional on whether `os.path.realpath.__code__.co_filename` is an absolute path.

`os.path` is loaded from `Spyder.app/Contents/Resources/lib/python39.zip/posixpath.pyc`, but `os.path.realpath.__code__.co_filename` is just `posixpath.pyc` which is not an absolute path, resulting in `pydev_log` printing its critical message.

# Conclusions
First, while I don't know exactly why the critical message is displayed in the IPython console in some circumstances and not others, I believe that #14025 _robustly_ revealed the issue by improving the stdout/err capture. I have tested whether `os.path.realpath.__code__.co_filename` is an absolute path for several of our releases and local builds under various versions, etc; they all indicate that it is not an absolute path and therefore trigger the `pydev_log.critical` message. I believe the only explanations for the different IPython console results are: (a) some idiosyncrasy with `pydev_log` (log level?); or (b) some idiosyncrasy with how Spyder captures and/or displays these messages. Perhaps it is (b) and #14025 removes the idiosyncracy, thus ensuring _consistent_ capture of these log messages. The evidence presented in #16795 supports this conclusion.

Second, while anything imported from modules in `Spyder.app/Contents/Resources/lib/python39.zip/` will likely have the same absolute path status, I think this is benign. The macOS Spyder application has been running, so clearly these packages (and their modules, submodules, functions, etc.) have been found, so no problem there. The message originates from `debugpy` and indicates that breakpoints may not be found by the debugger. Since anything imported from `Spyder.app/Contents/Resources/lib/python39.zip/` will be byte-compiled code, it is impossible for the user to set breakpoints in these codes, so the warning should not apply to us. Note that this issue should not affect breakpoint discovery in anything outside the zip folder, including a user's source code.

I'll propose a solution in a forthcoming comment.
> Perhaps it is (b) and #14025 removes the idiosyncracy, thus ensuring consistent capture of these log messages. The evidence presented in #16795 supports this conclusion.

Yeah, I think that's definitely the case.

> I'll propose a solution in a forthcoming comment.

Thanks @mrclary!
As I see it, we have two objectives:
* Ensure that this particular `pydev_log` message does not cause the installer test to fail.
* Hide this particular `pydev_log` message from users since it is benign.

I do not see any way to ensure that `os.path.realpath.__code__.co_filename` is an absolute path, so I believe the only option is to filter it on our end. Note that after merging #14025, `pydev_log` message is displayed in the IPython console, but only briefly since the kernel did not actually fail to start. The following gif illustrates this behavior, and the still shot is provided for clarity.
![pydevd](https://user-images.githubusercontent.com/9618975/142343263-4e0b9831-61c7-46ed-9691-cf1ac97d316c.gif)
![image](https://user-images.githubusercontent.com/9618975/142343585-dc5ba971-f12b-4a6d-a59b-7f1f3c5f5296.png)

@impact27, for the general case, should these types of logging messages from the stderr file be cleared from the IPython console like that?


Resolving patch:
diff --git a/spyder/plugins/ipythonconsole/widgets/client.py b/spyder/plugins/ipythonconsole/widgets/client.py
--- a/spyder/plugins/ipythonconsole/widgets/client.py
+++ b/spyder/plugins/ipythonconsole/widgets/client.py
@@ -485,9 +485,13 @@ def stop_button_click_handler(self):
     def show_kernel_error(self, error):
         """Show kernel initialization errors in infowidget."""
         self.error_text = error
-        if "issue1666807" not in error:
-            # TODO: remove the above if, see spyder-ide/spyder#16828
-            InstallerIPythonKernelError(error)
+
+        # Filter out benign errors
+        if "http://bugs.python.org/issue1666807" in error:
+            # See spyder-ide/spyder#16828
+            return
+
+        InstallerIPythonKernelError(error)
 
         # Replace end of line chars with <br>
         eol = sourcecode.get_eol_chars(error)


Test patch: