Repository: vispy/vispy

Issue:
Slow import
On my Windows workstation with >1200 of Python packages installed, `import vispy` takes about 3.4 seconds. 

98% of that time is spend at     
https://github.com/vispy/vispy/blob/ded293841c6438ab54af10c561f372155bb77edc/vispy/__init__.py#L26

On this system, more than half of `napari`'s import time is spent on determining the vispy version. 

Please consider not using `pkg_resources` during runtime.

Loosely related to https://github.com/napari/napari/pull/745

https://github.com/pypa/setuptools/issues/510


Discussion:
Oh, wow, great catch!
The reason pkg_resources is used is because we are using `setuptools_scm` to automatically determine the version number from the git history of the package. I figured this would be a performance hit, but I think we all (@kmuehlbauer @larsoner) decided it wouldn't be that bad and it is recommended ways of using setuptools_scm from their documentation. There is an option with setuptools_scm to write the version to a file (ex. `vispy/version.py`) but some maintainers were worried this would cause confusion or complications when new contributors were doing a dev installation and saw a new version.py file get created. After the last couple versions experience with setuptools_scm I'm starting to lean towards adding this version.py file the way I originally planned on doing it. I agree that this 3s import is just ridiculous.

I thought it would be good if you understood why `pkg_resources` was being used rather than assuming we were doing something just because we wanted to. This is the default way setuptools_scm works.

@kmuehlbauer @larsoner Feel free to chime in on concerns about using `vispy/version.py` (it is generated by setuptools_scm when you install the package from source).
To be fair, fixing this issue is not enough to speed up napari loading. There are other packages that are using `pkg_resources`: triangle, docutils, and sphinx.
@cgohlke for `napari` is it reasonable to nest the `triangle`, `docutils`, and `sphinx` imports? Or at least make it so that the submodules that use them are only loaded on demand?

Then `vispy` would still be the bottleneck. 

> I'm starting to lean towards adding this version.py file the way I originally planned on doing it. I agree that this 3s import is just ridiculous.

Agreed this is too much and we should avoid it. Writing `version.py` seems better in light of this overhead. Better to let `vispy` devs doing editable installs go through some confusion rather than all end users suffer through 3 sec imports.
@cgohlke Any idea, why `pkg_resources` needs that much time? 

For me it's only 0.2 sec to `time python -c "import pkg_resources"`. I'm going to guess it's because of the 1200+ packages installed on that system (see top comment), and `pkg_resources` doing some parsing of installed packages on import.
@larsoner I think the better test would be to do the `get_distribution` call for `vispy`. I think pkg_resources is having trouble finding the metadata for the vispy package among all the installed packages. Once it finds it then it should be parsing the metadata and pulling out the version number.
Ahh that would be better. 0.24 sec for `import vispy` for me, 0.20 for `python -c "import pkg_resources; pkg_resources.get_distribution('vispy')"`
> Any idea, why pkg_resources needs that much time?

On this system, `import pkg_resources` causes over 93,000 file system operations (CreateFile, CloseFile, QueryInformation...). 

> suffer through 3 sec imports

That is for a Python installation on a fast SSD with huge cache. Importing from a network drive takes in the order of 30 seconds.
Code for `get_distribution` if anyone is interested: https://bitbucket.org/pypa/pkg_resources/src/33e56f318f5086158de8bb2827acb55db2dbc153/pkg_resources.py?fileviewer=file-view-default#pkg_resources.py-368

Not that we are going to convince any changes that haven't been considered on the setuptools side. I'll just have to switch to using the version.py static file method. @cgohlke If I get this implemented in a pull request would you be able to test it on your system with the 1200 packages?

Also: https://github.com/pypa/setuptools/issues/510#issuecomment-463667124
> If I get this implemented in a pull request would you be able to test it on your system with the 1200 packages?

sure

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -178,7 +178,7 @@ def run(self):
 readme = open('README.rst', 'r').read()
 setup(
     name=name,
-    use_scm_version=True,
+    use_scm_version={'write_to': 'vispy/version.py'},
     author='Vispy contributors',
     author_email='vispy@googlegroups.com',
     license='(new) BSD',
@@ -209,7 +209,7 @@ def run(self):
         'jsdeps': NPM,
     },
     python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*',
-    install_requires=['numpy', 'freetype-py', 'setuptools'],
+    install_requires=['numpy', 'freetype-py'],
     setup_requires=['numpy', 'cython', 'setuptools_scm', 'setuptools_scm_git_archive'],
     extras_require={
         'ipython-static': ['ipython'],
@@ -223,7 +223,7 @@ def run(self):
         'wx': ['wxPython'],
         'doc': ['sphinx_bootstrap_theme', 'numpydoc'],
     },
-    packages=find_packages(),
+    packages=find_packages(exclude=['make']),
     ext_modules=cythonize(extensions),
     package_dir={'vispy': 'vispy'},
     data_files=[
diff --git a/vispy/__init__.py b/vispy/__init__.py
--- a/vispy/__init__.py
+++ b/vispy/__init__.py
@@ -18,13 +18,12 @@
 """
 
 from __future__ import division
-from pkg_resources import get_distribution, DistributionNotFound
 
 __all__ = ['use', 'sys_info', 'set_log_level', 'test']
 
 try:
-    __version__ = get_distribution(__name__).version
-except DistributionNotFound:
+    from .version import version as __version__  # noqa
+except ImportError:
     # package is not installed
     pass
 


Test patch:
diff --git a/vispy/util/tests/test_import.py b/vispy/util/tests/test_import.py
--- a/vispy/util/tests/test_import.py
+++ b/vispy/util/tests/test_import.py
@@ -16,7 +16,7 @@
 
 
 # minimum that will be imported when importing vispy
-_min_modules = ['vispy', 'vispy.util', 'vispy.ext', 'vispy.ipython', 'vispy.testing']
+_min_modules = ['vispy', 'vispy.util', 'vispy.ext', 'vispy.ipython', 'vispy.testing', 'vispy.version']
 
 
 def loaded_vispy_modules(import_module, depth=None, all_modules=False):