Repository: beetbox/beets

Issue:
Data files are unreadable when beets is installed as an egg
Following up from [a related Discourse thread](https://discourse.beets.io/t/error-running-from-source/739?u=arcresu), we noticed that beets isn't happy when it's installed as an egg. This is currently the default way beets is installed if you run `setup.py install` (not sure if #2083 changes this story).

#### Problem
In egg mode, the whole beets and beetsplug directory trees remain bundled in a zipfile and are not extracted. This causes problems when we try and read data files (aka package resources)  like `config_default.yaml`, because we currently build up the path to such files relative to the executing module's source file and `open()` the resulting path. Obviously this doesn't work of the file is inside a zipfile.

* Symptom: `ConfigError`s like `configuration error: verbose not found`.
 Caused by the failure to open `config_default.yaml` (aside: it would have been clearer if we got a `confit.ConfigReadError` rather than missing default values, but the loading of the default config file is inside a `if os.path.isfile(filename)` guard so it just silently skips the file when it doesn't exist even though beets depends on the default values being set).
Needs to be fixed in confuse.

* Symptom: `FileNotFoundError`s related to files like `lastgenre/genres.txt`.
Caused by failure to open other data files needed by plugins.
Needs to be fixed in beets.

#### Solutions

* Explicitly state that we don't support running beets as an egg. This seems unfortunate if it's the result of `setup.py install`.
* Try and fix the way data files are accessed. There is [`pkg_resources`](https://setuptools.readthedocs.io/en/latest/pkg_resources.html) from `setuptools` that has helpers to transparently handle this. We could refactor these data file accesses into a helper for plugins.
Data files are unreadable when beets is installed as an egg
Following up from [a related Discourse thread](https://discourse.beets.io/t/error-running-from-source/739?u=arcresu), we noticed that beets isn't happy when it's installed as an egg. This is currently the default way beets is installed if you run `setup.py install` (not sure if #2083 changes this story).

#### Problem
In egg mode, the whole beets and beetsplug directory trees remain bundled in a zipfile and are not extracted. This causes problems when we try and read data files (aka package resources)  like `config_default.yaml`, because we currently build up the path to such files relative to the executing module's source file and `open()` the resulting path. Obviously this doesn't work of the file is inside a zipfile.

* Symptom: `ConfigError`s like `configuration error: verbose not found`.
 Caused by the failure to open `config_default.yaml` (aside: it would have been clearer if we got a `confit.ConfigReadError` rather than missing default values, but the loading of the default config file is inside a `if os.path.isfile(filename)` guard so it just silently skips the file when it doesn't exist even though beets depends on the default values being set).
Needs to be fixed in confuse.

* Symptom: `FileNotFoundError`s related to files like `lastgenre/genres.txt`.
Caused by failure to open other data files needed by plugins.
Needs to be fixed in beets.

#### Solutions

* Explicitly state that we don't support running beets as an egg. This seems unfortunate if it's the result of `setup.py install`.
* Try and fix the way data files are accessed. There is [`pkg_resources`](https://setuptools.readthedocs.io/en/latest/pkg_resources.html) from `setuptools` that has helpers to transparently handle this. We could refactor these data file accesses into a helper for plugins.


Discussion:
Indeed; I agree those are basically the two options! In my view, Python Eggs are a relic of the past—they served an important role for their time, but most of the problems they solve these days are better solved by the PEP 427 Wheel format. Here's a quick [comparison between Egg and Wheel](https://packaging.python.org/discussions/wheel-vs-egg/) in the official docs.

That said, it does seem like `python setup.py install` installs an egg by default, which is too bad. I am wary of introducing a *runtime* dependency on setuptools, via its `pkg_resources` module—we currently only need it for installation, which seems like the Way It Should Be. (That is, if you were to install beets from a Wheel, you could do so without ever needing setuptools.)

It looks like you can instruct setuptools to avoid installing as an egg using [the `zip_safe` option](https://setuptools.readthedocs.io/en/latest/setuptools.html#setting-the-zip-safe-flag). Maybe we should try setting `zip_safe=False` and see if that forces `python setup.py install` to use an ordinary source installation?
It looks like it's probably not worth supporting a runtime beets egg for the reasons you mention (wheels replacing eggs and the runtime dependency on setuptools). The [Hacking](https://github.com/beetbox/beets/wiki/Hacking) page currently includes an egg URL in the local setup instructions that might need changing.

I'm happy to play around with `zip_safe` and move us off the egg-by-default setup but I wonder if @jackwilsdon has any thoughts since his proposed changes in #2083 involve egg-related stuff.

Whatever we do, I think Confuse might need a little attention related to this scenario. I'll have a look into that separately and move that discussion to its repo.
Indeed; I agree those are basically the two options! In my view, Python Eggs are a relic of the past—they served an important role for their time, but most of the problems they solve these days are better solved by the PEP 427 Wheel format. Here's a quick [comparison between Egg and Wheel](https://packaging.python.org/discussions/wheel-vs-egg/) in the official docs.

That said, it does seem like `python setup.py install` installs an egg by default, which is too bad. I am wary of introducing a *runtime* dependency on setuptools, via its `pkg_resources` module—we currently only need it for installation, which seems like the Way It Should Be. (That is, if you were to install beets from a Wheel, you could do so without ever needing setuptools.)

It looks like you can instruct setuptools to avoid installing as an egg using [the `zip_safe` option](https://setuptools.readthedocs.io/en/latest/setuptools.html#setting-the-zip-safe-flag). Maybe we should try setting `zip_safe=False` and see if that forces `python setup.py install` to use an ordinary source installation?
It looks like it's probably not worth supporting a runtime beets egg for the reasons you mention (wheels replacing eggs and the runtime dependency on setuptools). The [Hacking](https://github.com/beetbox/beets/wiki/Hacking) page currently includes an egg URL in the local setup instructions that might need changing.

I'm happy to play around with `zip_safe` and move us off the egg-by-default setup but I wonder if @jackwilsdon has any thoughts since his proposed changes in #2083 involve egg-related stuff.

Whatever we do, I think Confuse might need a little attention related to this scenario. I'll have a look into that separately and move that discussion to its repo.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -65,6 +65,7 @@ def build_manpages():
     platforms='ALL',
     long_description=_read('README.rst'),
     test_suite='test.testall.suite',
+    zip_safe=False,
     include_package_data=True,  # Install plugin resources.
 
     packages=[


Test patch: