Repository: napari/napari

Issue:
Unable to install any packages/plugins in current bundle
## 🐛 Bug

Unable to install any packages/plugins in current bundle.

## To Reproduce

Steps to reproduce the behavior:

1. Download recent bundle build: https://github.com/napari/napari/actions/runs/988280586
2. Try to install any plugins or other packages in plugins > Install/Remove plugins...
3. Get the error: `error: unrecognized arguments: --upgrade`

Latest version of pip has not actually removed `--upgrade` so it's unclear to me what is going on. For reference the code that is failing is here:

https://github.com/napari/napari/blob/3aef5a5e6ab6c4fb4a83202040f1234ee4aca90b/napari/_qt/dialogs/qt_plugin_dialog.py#L82-L93


Discussion:
@goanpeca maybe you can take a look at this when you get a chance
Ok, the bug goes all the way back to 0.4.7 (doesn't work). 0.4.6 works. That points to the ironically-named #2436 as the likely culprit. 😂 
I should say I'm running all this on macOS 10.15.7 Catalina.
I also see same thing on macOS 10.15.7 Catalina with nightly build
What is the pip version that comes with the bundle 🤔  ?
> What is the pip version that comes with the bundle 🤔 ?

Difficult to answer because of #2964 😂 
> That points to the ironically-named #2436 as the likely culprit. 😂

Having said that, I don't see anything there that would cause this.... Except it corresponds to a briefcase update... 😬 😬 😬 
@goanpeca Given the error message I don't think pip is even being called. Perhaps it is not even being packaged???
~How can I download the artifact?~

Nevermind, the browser was not scrolling down


Anyway, for reference 😂 

(1) Each release has artifacts attached: https://github.com/napari/napari/releases . With releases you have to click on "Assets" to see them.

(2) We have a nightly/on-push-to-master action that builds them: https://github.com/napari/napari/actions/workflows/make_bundle.yml
My hypothesis is that the briefcase update in #2436 stopped packaging pip. However I don't know how to test it right now...
never mind: both 0.4.6 and 0.4.7 contain pip 21.0.1 in `$APP/Contents/Resources/app_packages`.
The command that is running is this `/Applications/napari.app/Contents/MacOS/napari -m pip install --upgrade ...` so it is running napari not python.

so it seem `sys.executable` is pointing to `/Applications/napari.app/Contents/MacOS/napari` and not to the bundled python.

I do not see a `python` executable anywhere on the bundled app contents.
Yeah we are not yet signing our bundles. If you have experience with that @goanpeca a fix would be much appreciated!

But as an aside: screw Apple and their Move to Trash button.

> The command that is running is this

how did you find that @goanpeca? Nice sleuthing! I wonder then if actually what changed is how napari executes Python scripts... @tlambert03 played with that a fair bit...

Can you check whether the same thing happens in 0.4.6?
I modified the bundled files adding a `QMessageBox` and ran that on the install method inside .

`/Applications/napari.app/Contents/Resources/app/napari/_qt/dialogs/qt_plugin_dialog.py`

```python
    def install(self, pkg_list: Sequence[str]):
        cmd = ['-m', 'pip', 'install', '--upgrade']
        if running_as_bundled_app() and sys.platform.startswith('linux'):
            cmd += [
                '--no-warn-script-location',
                '--prefix',
                user_plugin_dir(),
            ]

        from qtpy.QtWidgets import QMessageBox
        QMessageBox.information(None, "test", " ".join([self.process.program()] + cmd + list(pkg_list)))
        self.process.setArguments(cmd + list(pkg_list))
        if self._output_widget:
            self._output_widget.clear()
        self.process.start()
```


>But as an aside: screw Apple and their Move to Trash button.

😆 
>Can you check whether the same thing happens in 0.4.6?

Will do 
> I wonder then if actually what changed is how napari executes Python scripts... @tlambert03 played with that a fair bit...

looking at the [release notes](https://github.com/napari/napari/releases/tag/v0.4.7) doesn't turn up anything suspicious/related to this...
@jni  on 0.4.6 it looks like, so it may be a briefcase thing that changed.

<img width="532" alt="Screen Shot 2021-06-30 at 11 53 24 PM" src="https://user-images.githubusercontent.com/3627835/124066500-5d968c00-d9fe-11eb-8905-96e88a9be6e1.png">

There is no `support/bin/python` available on the 0.4.9 bundle .
Ok, so it was the briefcase upgrade after all. Hmm. That'll be fun to fix. LOL
In the meantime we should bin to the latest working version of it ?
good Q. I think the update was required to e.g. have the version not be 0.0.1 in the bundle (see what happens when you open 0.4.6). Maybe there is an alternate strategy to achieve what #2436 was doing. CC @ziyangczi 
@jni I asked Russell over the discord beeware channel

He said the following:

>Yes - a month or two back we made a modification to the way we package Python apps to get around some limitations with sandboxing in Big Sur.

>Big Sur wasn't very happy with the general approach of using a shell script as the "app", launching python as a second executable

>So - we now package a bootstrap app that is very similar to what happens on iOS - a C program that doesn't do much more than "start the Python interpreter"

>The downside is that fork-based subprocesses don't work any more, because there isn't a python interpreter (by default), and sys.executable is now hard coded to start a specific app.
(threads still work - it's only forks that are a problem)

>Option 1: Modify the stub app so it behaves like a "normal" python executable when it's called with python-like arguments (like -m), but behaves like the existing stub when it is given no arguments.
That would be the most generic solution - and it would also mean that fork-based processing would work again, because sys.executable would be a viable Python interpreter.

>Option 2: A custom Python support package that includes a python executable. We're currently explicitly removing bin/python from the support package because it isn't needed when the stub exists; but if you include it, you should be able to call it.

>(This very use case was raised earlier today by SpencerNewton - so there might be an argument to include the python binary as part of the default support package, and strip it as part of briefcase, with that stripping being an option that is enabled by default, but could be disabled by a project like Napari)

>Option 3: would be to work out how to invoke pip as an API - I know that's something that PyPA strictly advises against, but... it might still work :slight_smile:

So we have something to work with :) 
> Yeah we are not yet signing our bundles. If you have experience with that @goanpeca a fix would be much appreciated!

Quick reminder here, it's not that we're not signing bundles (we are signing bundles), it's that we aren't signing with a proper (paid) Apple developer id.  

Still waiting for czi to buy us one of those ;)
> @jni I asked Russell over the discord beeware channel

That explains it! (I was trying to find python last night and wondering where it went)

That's a big deal for us... and perhaps reason to reconsider conda constructor or a variant like what I played with early on https://github.com/tlambert03/conda-bundler
@goanpeca thanks for the investigation and reaching out to Russell! It looks like Option 1 and 2 both require dev on the briefcase side and would take at least a week to get going. So for the immediate term perhaps we should either:

- revert #2436 and/or pin briefcase (ie I'm not sure how much of #2436 needs to be reverted), or
- Use pip as a library temporarily :grimacing:. For reference the links about this are [the official warning](https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program), and [this SO answer](https://stackoverflow.com/a/15950647/224254), ie `from pip._internal import main as pip_main; pip_main(['install', package_name])` (`pip.main` has been removed.)

By the way, between Option 1 and 2 I prefer option 1 because I don't think we want the napari bundle to break if a plugin uses `multiprocessing`. Having said that, I wonder whether in Option 2 `sys.executable` can be patched to point to the `python` binary and that would fix multiprocessing.
> * I'm not sure how much of [Fix packaging #2436](https://github.com/napari/napari/pull/2436) needs to be reverted), or

from a quick look, the only consequential thing in #2436 is the `if: runner.os == 'macOS' run: python -m pip install briefcase>=0.3.5` line.  (everything else is mostly "while you're in there" kind of stuff.)  So I'd vote for reverting that, and pinning to an earlier version.  
I'm not opposed to option 2, but only as a strictly temporary workaround, and since it will likely take a little testing and doing to make it work, I'm not sure it's worth it for a temp wrokaround.
@tlambert03 it looks like we would reopen #2389 if we revert #2436, but that is preferable to not being able to install plugins imho. Do we all agree?
yeah, that seems to be the tradeoff.  fine with me
> good Q. I think the update was required to e.g. have the version not be 0.0.1 in the bundle (see what happens when you open 0.4.6). Maybe there is an alternate strategy to achieve what #2436 was doing. CC @ziyangczi

at the time being, I didn't find viable alternatives after trying things I can think of. I can look into this again a bit later this week (currently busy with other things)

Also no objection pining down the version, I can try following up with briefcase again and see if they have been making any progress making the latest version working for us. Unless @goanpeca if you happen to know how can we address #2389 without changing the start script in the application?
This issue has been mentioned on **Image.sc Forum**. There might be relevant details there:

https://forum.image.sc/t/error-installing-cellpose-plugin-for-napari/55589/2

as noted in #3097, pinning briefcase to 0.3.1 in https://github.com/napari/napari/pull/2980 did _not_ actually change anything about this problem, since the newer support package (with static libs) is still being used.

we'll need to pin the support package as well, and since [3.8-b5](https://github.com/beeware/Python-Apple-support/releases/tag/3.8-b5) states that it's the first version in static lib format, rather than binary format, we probably need to pin earlier (see [here](https://briefcase.readthedocs.io/en/latest/reference/configuration.html?highlight=support_package#support-revision)):

```toml
[tool.briefcase.app.napari.macOS]
# 3.8-b4 was the last macOS support package in binary format, rather than static lib
# for python 3.9 use 3.9-b1
support_revision = 'b4'
```

@goanpeca, do you have time to fix this?  see #3097 ... where this is still popping up
Pinging @jaimergp

Resolving patch:
diff --git a/bundle.py b/bundle.py
--- a/bundle.py
+++ b/bundle.py
@@ -90,9 +90,28 @@ def patched_toml():
 
     print("patching pyproject.toml to version: ", VERSION)
     print(
-        "patching pyproject.toml requirements to : \n",
-        "\n".join(toml['tool']['briefcase']['app'][APP]['requires']),
+        "patching pyproject.toml requirements to:",
+        *toml['tool']['briefcase']['app'][APP]['requires'],
+        sep="\n ",
     )
+
+    if MACOS:
+        # Workaround https://github.com/napari/napari/issues/2965
+        # Pin revisions to releases _before_ they switched to static libs
+        revision = {
+            (3, 6): 'b11',
+            (3, 7): 'b5',
+            (3, 8): 'b4',
+            (3, 9): 'b1',
+        }[sys.version_info[:2]]
+        app_table = toml['tool']['briefcase']['app'][APP]
+        app_table.add('macOS', tomlkit.table())
+        app_table['macOS']['support_revision'] = revision
+        print(
+            "patching pyproject.toml to pin support package to revision:",
+            revision,
+        )
+
     with open(PYPROJECT_TOML, 'w') as f:
         f.write(tomlkit.dumps(toml))
 
@@ -169,6 +188,20 @@ def patch_wxs():
             print("patched pythonw.exe -> python.exe")
 
 
+def patch_python_lib_location():
+    # must run after briefcase create
+    support = os.path.join(
+        BUILD_DIR, APP, APP + ".app", "Contents", "Resources", "Support"
+    )
+    python_resources = os.path.join(support, "Python", "Resources")
+    os.makedirs(python_resources, exist_ok=True)
+    for subdir in ("bin", "lib"):
+        orig = os.path.join(support, subdir)
+        dest = os.path.join(python_resources, subdir)
+        os.symlink("../../" + subdir, dest)
+        print("symlinking", orig, "to", dest)
+
+
 def make_zip():
     import glob
     import zipfile
@@ -213,7 +246,10 @@ def bundle():
 
         if WINDOWS:
             patch_wxs()
+        elif MACOS:
+            patch_python_lib_location()
 
+        sys.exit(0)
         # build
         cmd = ['briefcase', 'build'] + (['--no-docker'] if LINUX else [])
         subprocess.check_call(cmd)
diff --git a/napari/_qt/dialogs/qt_plugin_dialog.py b/napari/_qt/dialogs/qt_plugin_dialog.py
--- a/napari/_qt/dialogs/qt_plugin_dialog.py
+++ b/napari/_qt/dialogs/qt_plugin_dialog.py
@@ -81,7 +81,7 @@ def _create_process(
         if installer != "pip":
             process.setProgram(installer)
         else:
-            process.setProgram(sys.executable)
+            process.setProgram(self._sys_executable_or_bundled_python())
 
         process.setProcessChannelMode(QProcess.MergedChannels)
         process.readyReadStandardOutput.connect(
@@ -106,6 +106,16 @@ def _create_process(
         )
         return process
 
+    def _sys_executable_or_bundled_python(self):
+        # Note: is_bundled_app() returns False even if using a Briefcase bundle...
+        # Workaround: see if sys.executable is set to something something napari on Mac
+        if sys.executable.endswith("napari") and sys.platform == 'darwin':
+            # sys.prefix should be <napari.app>/Contents/Resources/Support/Python/Resources
+            python = os.path.join(sys.prefix, "bin", "python3")
+            if os.path.isfile(python):
+                return python
+        return sys.executable
+
     def set_output_widget(self, output_widget: QTextEdit):
         if output_widget:
             self._output_widget = output_widget


Test patch: