Repository: fedora-infra/bodhi

Issue:
Error in setup.py in bodhi-client-3.3.0
Got this error by pip install.

```
pip install "bodhi-client>=2.0,<4"
Collecting bodhi-client<4,>=2.0
  Downloading https://files.pythonhosted.org/packages/d3/8e/ce082026aca1e337a317c44c44c4834c7d6f7de05948c18f427ea1411026/bodhi-client-3.3.0.tar.gz
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/bodhi.egg-info
    writing pip-egg-info/bodhi.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/bodhi.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/bodhi.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/bodhi.egg-info/SOURCES.txt'
    package init file 'bodhi/__init__.py' not found (or not a regular file)
    reading manifest file 'pip-egg-info/bodhi.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/bodhi.egg-info/SOURCES.txt'
    running egg_info
    creating pip-egg-info/bodhi_client.egg-info
    writing requirements to pip-egg-info/bodhi_client.egg-info/requires.txt
    writing pip-egg-info/bodhi_client.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/bodhi_client.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/bodhi_client.egg-info/dependency_links.txt
    writing entry points to pip-egg-info/bodhi_client.egg-info/entry_points.txt
    writing manifest file 'pip-egg-info/bodhi_client.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/bodhi_client.egg-info/SOURCES.txt'
    reading manifest template 'CLIENT_MANIFEST.in'
    writing manifest file 'pip-egg-info/bodhi_client.egg-info/SOURCES.txt'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-quDnNB/bodhi-client/setup.py", line 127, in <module>
        server_packages.remove('bodhi')
    ValueError: list.remove(x): x not in list
```


Discussion:
This also happens in `bodhi-client-2.9.0`
I tried to add

```python
if 'bodhi' in server_packages:
    server_packages.remove('bodhi')
```

The crash is fixed but it causes another problem, e.g. `pip install bodhi-client-3.7.0.tar.gz`, `bodhi-server` is to be installed and only it, no `bodhi-client` is installed.

How about use `extra_require` for `bodhi-server` and `bodhi-client`? Or split `bodhi/client` and `bodhi/server` into separate Python packages with their own `setup.py`?
May I ask is there any reason to call `setup` three times to create `bodhi`, `bodhi-client` and `bodhi-server` tarballs separately?

I'm also thinking of a new layout of source code in order to install bodhi-client and bodhi-server easily that could fix this issue. Initial idea is:

* client, server and bodhi common files move into separate subpackages with their own package files, e.g. `setup.py`, `MANIFEST.in`, etc.

* an alternative one is to distribute bodhi source tarball by including everything inside the source tree, which allows developers to get a copy of complete source code from tarball. As a result, there is only one bodhi package which installed client, server and common files together, whose dependent packages are listed as extra list individually. So, client could be installed by `bodhi[client]`, and server could be installed by `bodhi[server]`.

What do you think?
Yeah I've been thinking about changing this for a while. The way I'm using ```setup.py``` is certainly not the way it's intended to be used, and it does cause some issues as you've noted.

@tkdchen can you explain that second proposal a bit more? I'm not sure I follow.

Giving them each their own subpackage is reasonable I think.
@bowlofeggs The second one is to use setuptools `extras_require`. Personally, I prefer the first proposal.
I see the same error when trying to install `bodhi-client-3.11.0` from pypi:
```
$ virtualenv envtest
Using base prefix '/usr'
New python executable in /home/kparal/devel/taskotron/libtaskotron/envtest/bin/python3
Also creating executable in /home/kparal/devel/taskotron/libtaskotron/envtest/bin/python
Installing setuptools, pip, wheel...
done.

$ source envtest/bin/activate

$ pip install bodhi-client
Collecting bodhi-client
  Downloading https://files.pythonhosted.org/packages/97/f7/3e764782773915bd8a6254663c6d3f0d28a2954a78f7bdeb9a04797c7d54/bodhi-client-3.11.0.tar.gz
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/bodhi.egg-info
    writing pip-egg-info/bodhi.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/bodhi.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/bodhi.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/bodhi.egg-info/SOURCES.txt'
    package init file 'bodhi/__init__.py' not found (or not a regular file)
    reading manifest file 'pip-egg-info/bodhi.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/bodhi.egg-info/SOURCES.txt'
    running egg_info
    creating pip-egg-info/bodhi_client.egg-info
    writing pip-egg-info/bodhi_client.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/bodhi_client.egg-info/dependency_links.txt
    writing entry points to pip-egg-info/bodhi_client.egg-info/entry_points.txt
    writing requirements to pip-egg-info/bodhi_client.egg-info/requires.txt
    writing top-level names to pip-egg-info/bodhi_client.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/bodhi_client.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/bodhi_client.egg-info/SOURCES.txt'
    reading manifest template 'CLIENT_MANIFEST.in'
    writing manifest file 'pip-egg-info/bodhi_client.egg-info/SOURCES.txt'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-07z_afee/bodhi-client/setup.py", line 123, in <module>
        server_packages.remove('bodhi')
    ValueError: list.remove(x): x not in list
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-07z_afee/bodhi-client/
```
This is a blocker for us in Taskotron test suite when trying to transition from `python-fedora` to `bodhi-client`. Can this be fixed soon, please?
In our case, I **"** solved **"** this by skipping the test in tox and running it when invoked directly.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,8 +1,11 @@
+from multiprocessing import Process
 import os
 
 from setuptools import setup, find_packages
 import setuptools.command.egg_info
 
+from bodhi import __version__ as VERSION
+
 
 def get_requirements(requirements_file='requirements.txt'):
     """
@@ -38,7 +41,6 @@ def get_requirements(requirements_file='requirements.txt'):
 
 here = os.path.abspath(os.path.dirname(__file__))
 README = open(os.path.join(here, 'README.rst')).read()
-VERSION = '5.4.0'
 
 
 # Possible options are at https://pypi.python.org/pypi?%3Aaction=list_classifiers
@@ -58,27 +60,39 @@ def get_requirements(requirements_file='requirements.txt'):
 PLATFORMS = ['Fedora', 'GNU/Linux']
 URL = 'https://github.com/fedora-infra/bodhi'
 
+client_pkgs = find_packages(
+    include=['bodhi.client', 'bodhi.client.*'],
+)
 
-setuptools.command.egg_info.manifest_maker.template = 'BODHI_MANIFEST.in'
-
-
-setup(
-    name='bodhi',
-    version=VERSION,
-    description='bodhi common package',
-    long_description=README,
-    classifiers=CLASSIFIERS,
-    license=LICENSE,
-    maintainer=MAINTAINER,
-    maintainer_email=MAINTAINER_EMAIL,
-    platforms=PLATFORMS,
-    url=URL,
-    keywords='fedora',
-    packages=['bodhi'],
-    include_package_data=True,
-    zip_safe=False,
-    install_requires=[],
-    tests_require=[
+messages_pkgs = find_packages(
+    include=['bodhi.messages', 'bodhi.messages.*'],
+)
+
+server_pkgs = find_packages(
+    include=['bodhi.server', 'bodhi.server.*'],
+)
+
+base_setup = {
+    'version': VERSION,
+    'long_description': README,
+    'classifiers': CLASSIFIERS,
+    'license': LICENSE,
+    'maintainer': MAINTAINER,
+    'maintainer_email': MAINTAINER_EMAIL,
+    'platforms': PLATFORMS,
+    'url': URL,
+    'zip_safe': False,
+}
+
+bodhi_setup = {
+    'manifest': 'BODHI_MANIFEST.in',
+    'name': 'bodhi',
+    'description': 'bodhi common package',
+    'keywords': 'fedora',
+    'packages': ['bodhi'],
+    'include_package_data': True,
+    'install_requires': [],
+    'tests_require': [
         'flake8',
         'pytest',
         'pytest-cov',
@@ -88,70 +102,42 @@ def get_requirements(requirements_file='requirements.txt'):
         'psycopg2',
         'requests',
     ],
-)
-
-
-setuptools.command.egg_info.manifest_maker.template = 'CLIENT_MANIFEST.in'
-
-
-setup(
-    name='bodhi-client',
-    version=VERSION,
-    description='bodhi client',
-    long_description=README,
-    classifiers=CLASSIFIERS,
-    license=LICENSE,
-    maintainer=MAINTAINER,
-    maintainer_email=MAINTAINER_EMAIL,
-    platforms=PLATFORMS,
-    url=URL,
-    keywords='fedora',
-    packages=['bodhi.client'],
-    include_package_data=False,
-    zip_safe=False,
-    install_requires=['click', 'python-fedora >= 0.9.0'],
-    entry_points="""\
-    [console_scripts]
-    bodhi = bodhi.client:cli
-    """)
-
-
-setuptools.command.egg_info.manifest_maker.template = 'MESSAGES_MANIFEST.in'
-
-
-setup(
-    name="bodhi-messages",
-    version=VERSION,
-    description="JSON schema for messages sent by Bodhi",
-    long_description=('Bodhi Messages\n==============\n\n This package contains the schema for '
-                      'messages published by Bodhi.'),
-    url="https://github.com/fedora-infra/bodhi/",
-    # Possible options are at https://pypi.python.org/pypi?%3Aaction=list_classifiers
-    classifiers=[
-        "License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)",
-        "Operating System :: POSIX :: Linux",
-        "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.6",
-        "Programming Language :: Python :: 3.7",
-    ],
-    license="GPLv2+",
-    maintainer="Fedora Infrastructure Team",
-    maintainer_email="infrastructure@lists.fedoraproject.org",
-    platforms=["Fedora", "GNU/Linux"],
-    keywords=["fedora", "fedora-messaging"],
-    packages=find_packages(
-        exclude=['bodhi.client', 'bodhi.client.*', 'bodhi.server', 'bodhi.server.*', 'bodhi.tests',
-                 'bodhi.tests.*']),
-    include_package_data=True,
-    zip_safe=False,
-    install_requires=["fedora_messaging"],
-    test_suite="bodhi.tests.messages",
-    entry_points={
+}
+
+client_setup = {
+    'manifest': 'CLIENT_MANIFEST.in',
+    'name': 'bodhi-client',
+    'description': 'bodhi client',
+    'keywords': 'fedora',
+    'packages': client_pkgs,
+    'include_package_data': False,
+    'install_requires': ['click', 'python-fedora >= 0.9.0', 'koji'],
+    'entry_points': '''
+        [console_scripts]
+        bodhi = bodhi.client:cli
+    ''',
+}
+
+messages_setup = {
+    'manifest': 'MESSAGES_MANIFEST.in',
+    'name': 'bodhi-messages',
+    'description': 'JSON schema for messages sent by Bodhi',
+    'long_description': (
+        'Bodhi Messages\n==============\n\n This package contains the schema for '
+        'messages published by Bodhi.'
+    ),
+    'keywords': ["fedora", "fedora-messaging"],
+    'packages': messages_pkgs,
+    'include_package_data': True,
+    'install_requires': ['bodhi', 'fedora_messaging'],
+    'test_suite': 'bodhi.tests.messages',
+    'entry_points': {
         "fedora.messages": [
             (
                 "bodhi.buildroot_override.tag.v1="
                 "bodhi.messages.schemas.buildroot_override:BuildrootOverrideTagV1"
-            ), (
+            ),
+            (
                 "bodhi.buildroot_override.untag.v1="
                 "bodhi.messages.schemas.buildroot_override:BuildrootOverrideUntagV1"
             ),
@@ -185,59 +171,52 @@ def get_requirements(requirements_file='requirements.txt'):
             (
                 "bodhi.update.request.obsolete.v1="
                 "bodhi.messages.schemas.update:UpdateRequestObsoleteV1"
-            ), (
+            ),
+            (
                 "bodhi.update.requirements_met.stable.v1="
                 "bodhi.messages.schemas.update:UpdateRequirementsMetStableV1"
             ),
         ]
     },
-)
-
-
-setuptools.command.egg_info.manifest_maker.template = 'SERVER_MANIFEST.in'
-# Due to https://github.com/pypa/setuptools/issues/808, we need to include the bodhi superpackage
-# and then remove it if we want find_packages() to find the bodhi.server package and its
-# subpackages without including the bodhi top level package.
-server_packages = find_packages(
-    exclude=['bodhi.client', 'bodhi.client.*', 'bodhi.messages', 'bodhi.messages.*', 'bodhi.tests',
-             'bodhi.tests.*'])
-server_packages.remove('bodhi')
-
-
-setup(
-    name='bodhi-server',
-    version=VERSION,
-    description='bodhi server',
-    long_description=README,
-    classifiers=CLASSIFIERS + [
-        'Framework :: Pyramid',
-        'Programming Language :: JavaScript',
-        "Topic :: Internet :: WWW/HTTP",
-        "Topic :: Internet :: WWW/HTTP :: WSGI :: Application"],
-    license=LICENSE,
-    maintainer=MAINTAINER,
-    maintainer_email=MAINTAINER_EMAIL,
-    platforms=PLATFORMS,
-    url=URL,
-    keywords='web fedora pyramid',
-    packages=server_packages,
-    include_package_data=True,
-    zip_safe=False,
-    install_requires=get_requirements(),
-    message_extractors={'.': []},
-    entry_points="""\
-    [paste.app_factory]
-    main = bodhi.server:main
-    [console_scripts]
-    initialize_bodhi_db = bodhi.server.scripts.initializedb:main
-    bodhi-push = bodhi.server.push:push
-    bodhi-untag-branched = bodhi.server.scripts.untag_branched:main
-    bodhi-skopeo-lite = bodhi.server.scripts.skopeo_lite:main
-    bodhi-sar = bodhi.server.scripts.sar:get_user_data
-    bodhi-shell = bodhi.server.scripts.bshell:get_bodhi_shell
-    bodhi-clean-old-composes = bodhi.server.scripts.compat:clean_old_composes
-    bodhi-expire-overrides = bodhi.server.scripts.compat:expire_overrides
-    bodhi-approve-testing = bodhi.server.scripts.compat:approve_testing
-    bodhi-check-policies = bodhi.server.scripts.compat:check_policies
-    """,
-    paster_plugins=['pyramid'])
+}
+
+server_setup = {
+    'manifest': 'SERVER_MANIFEST.in',
+    'name': 'bodhi-server',
+    'description': 'bodhi server',
+    'classifiers': (
+        CLASSIFIERS + [
+            'Framework :: Pyramid',
+            'Programming Language :: JavaScript',
+            "Topic :: Internet :: WWW/HTTP",
+            "Topic :: Internet :: WWW/HTTP :: WSGI :: Application",
+        ]
+    ),
+    'keywords': ['web', 'fedora', 'pyramid'],
+    'packages': server_pkgs,
+    'include_package_data': False,
+    'install_requires': get_requirements(),
+    'message_extractors': {'.': []},
+    'entry_points': '''
+        [paste.app_factory]
+        main = bodhi.server:main
+        [console_scripts]
+        initialize_bodhi_db = bodhi.server.scripts.initializedb:main
+        bodhi-push = bodhi.server.push:push
+        bodhi-untag-branched = bodhi.server.scripts.untag_branched:main
+        bodhi-skopeo-lite = bodhi.server.scripts.skopeo_lite:main
+        bodhi-sar = bodhi.server.scripts.sar:get_user_data
+        bodhi-shell = bodhi.server.scripts.bshell:get_bodhi_shell
+        bodhi-clean-old-composes = bodhi.server.scripts.compat:clean_old_composes
+        bodhi-expire-overrides = bodhi.server.scripts.compat:expire_overrides
+        bodhi-approve-testing = bodhi.server.scripts.compat:approve_testing
+        bodhi-check-policies = bodhi.server.scripts.compat:check_policies
+    ''',
+    'paster_plugins': ['pyramid'],
+}
+
+for s in [bodhi_setup, client_setup, messages_setup, server_setup, ]:
+    setuptools.command.egg_info.manifest_maker.template = s.pop('manifest')
+    p = Process(target=setup, kwargs={**base_setup, **s})
+    p.start()
+    p.join()


Test patch: