Repository: spyder-ide/spyder

Issue:
Spyder Mac app laggy in 5.2.0
## Problem Description

Hi,

Just letting you guys know that this problem is pretty pervasive. In an academic setting at my university, multiple labs across campus are realizing independently the turn for the worse Spyder is becoming, especially with the lag across Mac users. It's affecting a lot of undergrads who come through our labs too, and most now go to PyCharm as their python IDE for data science. I suspect most of these new users will never touch Spyder. I'm just the messenger here and letting you guys know that after seeing the trend for the last year, people are beginning to have an unfavorable opinion of Spyder across my university, including multiple PIs and professors that used to use Spyder in their courses. This problem seems not fixed yet, and despite implementations of bloated bells and whistles (like integration of Kite), the one thing that actually may affect user growth at its root is not being addressed. 


Discussion:
Hey @andyrevell, thanks a lot for reporting this problem and for voicing your concerns. We're aware of these issues and right now are working really hard to fix them.

A lot of improvements in this area were included in 5.1.5 (the latest version available in Anaconda), some more were added in 5.2.0 (our latest version), and right now we're working on even more improvements for 5.2.1 (see PR #16864).

Please try 5.2.0 by installing our [Mac app](https://github.com/spyder-ide/spyder/releases) and let us know how it behaves for you. Feedback from our community is very valuable and appreciated, and it's also crucial to understand how we should invest our time and efforts in the future.
5.1.5 mostly fixed the lag issues on Mac for me.  It was the best Spyder had run for me in at least a year, and much appreciated, so thanks for that. 

The lag seems to be back in full force with 5.2.0 though.  
Just as with versions < 5.1.5, the lag goes away if I disable the Python Language Server, but then there's no linting. 
> The lag seems to be back in full force with 5.2.0 though.

Hey @ghost711, thanks for chiming in! That's really bad news.

Some questions for you:

* Are you also using a Mac?
* If that's the case, are you on an M1 or Intel?
* How did you install Spyder?
* Could you upload your Spyder configuration to check what options are you using that cause the lag? You can find it in `~/.spyder-py3/config/spyder.ini` if you're on Mac.
@mrclary, @impact27, have you noticed any regressions when typing on 5.2.0?
Yep, I definitely see a regression...

![latency-regression](https://user-images.githubusercontent.com/9618975/144278993-f4b7a6cf-9441-4a3b-b07d-dd85a0c46611.gif)
Ok, that's really frustrating. I don't really know why the sluggishness reappeared again.

@mrclary, could you use `git bisect` between 5.1.5 and 5.2.0 to track down the faulty commit? As soon as you let me know about it, I'll try to fix the problem so we can include it in 5.2.1, to be released next week.
There does not seem to be a problem with the latest 5.x bootstrap, though. That is weird. I'll do some digging and report what I find.
I have seen this as well, but I think this is fixed by https://github.com/spyder-ide/spyder/pull/16864. It would however make sense to add a test to avoid this kind of regression and keep track of the evolution of spyder speed. I am not sure what the best way of doing that is though.
> There does not seem to be a problem with the latest 5.x bootstrap, though. That is weird. I'll do some digging and report what I find.

Oh, that's really weird! I mean, we haven't touched many things related to the editor after 5.2.0 was released, and the few shouldn't be related to performance (as far as I can tell).

> I have seen this as well, but I think this is fixed by #16864

Ok, then the best course of action is to wait until #16864 is ready to release 5.2.1

> I am not sure what the best way of doing that is though.

Yeah, me neither, that's why I haven't added a test for that yet. What if we use `timeit` as you did in #16864? I mean, we could use the value reported by CIs in that PR as our baseline (plus a small offset to account for differences), and then `assert` that new changes don't make that value higher.
> Ok, then the best course of action is to wait until #16884 is ready to release 5.2.1
> 
> > I am not sure what the best way of doing that is though.
> 
> Yeah, me neither, that's why I haven't added a test for that yet. What if we use `timeit` as you did in #16884? I mean, we could use the value reported by CIs in that PR as our baseline (plus a small offset to account for differences), and then `assert` that new changes don't make that value higher.

I think you meant #16864.
Sorry, that's right, I meant PR #16864.
The latest artifact from #16864 shows bad latency, but bootstrap from the latest commit 0b104325d29ef64f6d6ad80accebb30371f40381 on impact27/fast_spyder branch does _not_ show latency.

I suspect a difference in python packages between the build environment and the conda environment. I'll keep looking.
If you have a profiler run which shows latency I could have a look. If you run bootstrap with:
`python -m cProfile -o output.prof bootstrap.py`
And do a lot of whatever is laggy, I should be able to see it by analysing the output.prof file 
Update: I tried switching to Python 3.10, and now the lag appears to be gone!
EDIT: Sorry, that wasn't actually true. I had reenabled the PLS, but it didn't actually start linting again until I restarted it, after which the lag (and CPU usage) returned.

Answers to your previous questions:

> * Are you also using a Mac? 
Yes
> * If that's the case, are you on an M1 or Intel? 
Intel
> * How did you install Spyder?
.dmg installer.
> * Could you upload your Spyder configuration to check what options are you using that cause the lag? You can find it in `~/.spyder-py3/config/spyder.ini` if you're on Mac.
[spyder.ini.zip](https://github.com/spyder-ide/spyder/files/7636095/spyder.ini.zip)



@ccordoba12, I cannot check the artifacts from the last two merged merge requests #16915 and #16925. The MR were merged with mac installer failures:
```
Error: -01 15:32:43,990 [ERROR] [Installer] -> Missing dependencies
  # Mandatory:
  spyder_kernels >=2.2.0;<2.3.0 :  3.0.0.dev0 (NOK)
```
I've seen this before when attempting to merge into master instead of 5.x, so I don't know what is happening here. 
Okay, I found the issue. It is not a regression, rather #16446 did not completely resolve #14823 for the macOS installer because `jellyfish` did not get packaged with the installer. I consider this a failure on the installer since it copies the egg-info into the installer, so Spyder thinks it is installed (but it really isn't) and is not caught by the installer test.

I will submit a PR that (a) ensures that `jellyfish` is packaged with the installer and (b) ensures a more robust treatment of egg-info for the installer so that dependency tests are not fooled.
I did a `conda install jellyfish` and now the behavior seems better but not quite fixed. Is that what you expected?
I'm just doing the simplest test possible: Holding in the `d` key and watching the characters attempt to print at the repeat rate for the key.  
Before installing jellyfish: 
- Prints in bursts of decent speed, with periodic pauses (approx 0.5 secs every 2 secs).
After:
- Prints continuously, but seems to be at a slower rate than during the bursts before installing jellyfish.

In both cases, CPU goes nuts both during and for a long time after (several minutes).
@ghost711, installing jellyfish locally will not resolve the issue. It must be packaged inside the Spyder.app application, not part of an external environment.
Ok, I thought that might be the case.  The difference I saw might have just been due to unrelated system state then.

Resolving patch:
diff --git a/installers/macOS/setup.py b/installers/macOS/setup.py
--- a/installers/macOS/setup.py
+++ b/installers/macOS/setup.py
@@ -13,6 +13,8 @@
 
 import os
 import sys
+import shutil
+import pkg_resources
 from logging import getLogger, StreamHandler, Formatter
 from setuptools import setup
 
@@ -33,6 +35,9 @@
 
 sys.path.append(SPYREPO)
 
+from spyder import __version__ as SPYVER
+from spyder.config.base import MAC_APP_NAME
+
 # Python version
 PYVER = [sys.version_info.major, sys.version_info.minor,
          sys.version_info.micro]
@@ -125,12 +130,7 @@ def make_app_bundle(dist_dir, make_lite=False):
         NotADirectoryError: [Errno 20] Not a directory: '<path>/Resources/lib/
         python39.zip/textdistance/libraries.json'
     """
-    import shutil
-    import pkg_resources
-
-    from spyder import __version__ as SPYVER
     from spyder.config.utils import EDIT_FILETYPES, _get_extensions
-    from spyder.config.base import MAC_APP_NAME
 
     # Patch py2app for IPython help()
     py2app_file = pkg_resources.pkgutil.get_loader('py2app').get_filename()
@@ -155,6 +155,7 @@ def make_app_bundle(dist_dir, make_lite=False):
                 'textdistance', 'debugpy',
                 ]
     INCLUDES = ['_sitebuiltins',  # required for IPython help()
+                'jellyfish',
                 # required for sphinx
                 'sphinxcontrib.applehelp', 'sphinxcontrib.devhelp',
                 'sphinxcontrib.htmlhelp', 'sphinxcontrib.jsmath',
@@ -214,24 +215,57 @@ def make_app_bundle(dist_dir, make_lite=False):
         os.remove(app_script_path)
         os.remove(SPYLINK)
 
-    # Copy egg info from site-packages: fixes several pkg_resources issues
-    dest_dir = os.path.join(dist_dir, MAC_APP_NAME, 'Contents', 'Resources',
-                            'lib', f'python{PYVER[0]}.{PYVER[1]}')
+    # Copy egg info: fixes several pkg_resources issues
+    copy_egg_info(dist_dir)
+
+    return
+
+
+def copy_egg_info(dist_dir):
+    from zipfile import ZipFile
+
     pkg_resources.working_set.add_entry(SPYREPO)
+    egg_map = {}
     for dist in pkg_resources.working_set:
-        if (dist.egg_info is None or dist.key.startswith('pyobjc')
-                or dist.key in EXCLUDE_EGG):
-            logger.info(f'Skipping egg {dist.key}')
+        if dist.egg_info is None:
             continue
-        egg = os.path.basename(dist.egg_info)
-        dest = os.path.join(dest_dir, egg)
-        shutil.copytree(dist.egg_info, dest)
-        logger.info(f'Copied {egg}')
-
-    logger.info('App bundle complete.')
-
-    return
 
+        try:
+            tops = dist.get_metadata('top_level.txt').strip().split('\n')
+            for top in tops:
+                egg_map.update({top: dist.egg_info})
+        except FileNotFoundError:
+            egg_map.update({dist.project_name: dist.egg_info})
+
+    base_dir = os.path.join(dist_dir, MAC_APP_NAME,
+                            'Contents', 'Resources', 'lib')
+    pkg_dir = os.path.join(base_dir, 'python{}.{}'.format(*PYVER))
+    lib_dir = os.path.join(pkg_dir, 'lib-dynload')
+    zip_dir = os.path.join(base_dir, 'python{}{}.zip'.format(*PYVER))
+
+    pkgs = set(f.name for f in os.scandir(pkg_dir))
+    pkgs.update(f.name for f in os.scandir(lib_dir))
+    pkgs.update(item.split(os.sep)[0] for item in ZipFile(zip_dir).namelist())
+
+    eggs = set()
+    for pkg in pkgs:
+        top = pkg.split('.')[0]
+
+        egg = None
+        try:
+            dist = pkg_resources.get_distribution(top)
+            egg = dist.egg_info
+        except Exception:
+            egg = egg_map.get(top, None)
+
+        if egg is not None:
+            eggs.add(egg)
+
+    for egg in eggs:
+        egg_name = os.path.basename(egg)
+        dest = os.path.join(pkg_dir, egg_name)
+        shutil.copytree(egg, dest)
+        logger.info(f'Copied {egg_name}')
 
 def make_disk_image(dist_dir, make_lite=False):
     """
@@ -250,8 +284,6 @@ def make_disk_image(dist_dir, make_lite=False):
 
     from dmgbuild import build_dmg
     from dmgbuild.core import DMGError
-    from spyder import __version__ as SPYVER
-    from spyder.config.base import MAC_APP_NAME
 
     volume_name = '{}-{} Py-{}.{}.{}'.format(MAC_APP_NAME[:-4], SPYVER, *PYVER)
     dmgfile = os.path.join(dist_dir, 'Spyder')
@@ -295,6 +327,9 @@ def make_disk_image(dist_dir, make_lite=False):
                         default=False, help='Create disk image')
     parser.add_argument('-d', '--dist-dir', dest='dist_dir', default='dist',
                         help='Distribution directory; passed to py2app')
+    parser.add_argument('-b', '--bdist-base', dest='build_dir',
+                        default='build',
+                        help='Build directory; passed to py2app')
 
     args, rem = parser.parse_known_args()
 
@@ -302,8 +337,10 @@ def make_disk_image(dist_dir, make_lite=False):
     sys.argv = sys.argv[:1] + ['py2app'] + rem
 
     dist_dir = os.path.abspath(args.dist_dir)
+    build_dir = os.path.abspath(args.build_dir)
 
     if args.make_app:
+        shutil.rmtree(build_dir, ignore_errors=True)
         make_app_bundle(dist_dir, make_lite=args.make_lite)
     else:
         logger.info('Skipping app bundle.')
diff --git a/spyder/utils/installers.py b/spyder/utils/installers.py
--- a/spyder/utils/installers.py
+++ b/spyder/utils/installers.py
@@ -64,6 +64,8 @@ class InstallerInternalError(SpyderInstallerError):
     def _msg(self, msg):
         msg = 'Spyder internal error\n' + textwrap.indent(msg, '  ')
 
+        return msg
+
 
 class InstallerPylspError(SpyderInstallerError):
     """Error for PyLSP issues"""


Test patch: