Repository: dask/distributed

Issue:
distributed-1.15.0rc1 seems wrongly requiring "futures" from a Python-3.6 installation
Collecting futures (from distributed>=1.14; extra == "complete"->dask[complete]->-r C:\Winpython\basedir36


Discussion:
cc @rbubley
also, didn't you suggest in another dask issue 1870  that dask-0.13.0rc1 should require distributed-1.15.0rc1 ?
Yes, when I release them properly then I'll update the requirements
@stonebig, how are you installing dask-distributed to get this issue? If with pip, what version of pip are you using?
Î do:
````
pip download --dest  C:\the_packages  --pre dask
rem (and other packages)
````
then:`
````
pip install x.whl  --no-index --trusted-host=None  --find-links=C:\the_packages
````

It seems that your own rule creates the problem:
- in setup.py:
  - you add futures to requested list only if python <3,
  - but this list is pre-filled with requirements.txt,
- futures is in requirements.txt and not filtered by pip-9.0.1 and/or setuptools-32.3.1.

the problem should be reproducible in any Python-3.6 installation.

I didn't look in setuptools vs requirements filter specification. I didn't notice other packages using "requirements" to feed setup.py, until this problem.
or maybe you build the universal wheel from a Python-2.7 environnment, and the filter test is done at build time ?
I don't find where in the setuptools documentation it is specified that "install_requires=" does support 

> futures; python_version < '3.0'

http://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-dependencies

it may be a "requirement.txt" only feature, so a dedicated py3 wheel would be the solution 
It's in the pip docs (
http://pip.readthedocs.io/en/stable/reference/pip_install/#requirement-specifiers
)
On Sat, 31 Dec 2016 at 13:17, stonebig <notifications@github.com> wrote:

> I don't find where in the setuptools documentation it is specified that
> "install_requires=" does support
>
> futures; python_version < '3.0'
>
>
> http://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-dependencies
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/771#issuecomment-269864639>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AC4r043qPvXR0pLI-og3PJQM_hZSovNLks5rNlXugaJpZM4LYRB2>
> .
>
-- 





*Russ Bubley*

[image: Logo]



Innovation in Social Investment

    russ@i-for-change.co.uk

    020 8815 0864 <+442088150864>

*www.i-for-change.co.uk <http://www.i-for-change.co.uk/>*

    0797 119 5656 <+447971195656>

Follow us, like us.

[image: linkedin] <http://www.linkedin.com/company/i-for-change/>   [image:
twitter] <https://twitter.com/iforchangeUK>   [image: facebook]
<https://www.facebook.com/iforchangeUK>



*Get blog posts in your inbox
<http://www.i-for-change.co.uk/subscribe.html>*


This electronic communication is issued by i for change ltd., an Appointed
Representative of Sturgeon Ventures LLP who is authorised and regulated by
the Financial Conduct Authority FRN # 452811. Any unauthorised use or
interception of this e-mail is illegal. If this e-mail is not intended for
you, you may not copy, distribute or disclose the contents to anyone. Save
for bona fide company matters, i for change ltd. does not accept any
responsibility for the opinions expressed in this e-mail. This
communication is made available in the United Kingdom to professional
clients and eligible counterparties as defined by the FCA.

Persons who have received this communication should not rely upon it alone,
and should seek professional advice if necessary. Any attachments contained
in this email are in accordance with COBS 2.4 in the FCA Handbook on
www.FCA.org.uk <http://www.fca.org.uk/> (reliance on others). i for change
ltd. is taking a proportionate approach and has verified the attachments
and other email content to the best of our ability. However if you consider
the contents (including attachments) are material to the formation of a
contract and / or you are otherwise relying upon its accuracy, you may wish
to request a copy to be sent by normal mail.

i for change ltd. cannot guarantee the accuracy or completeness of this
e-mail as information could be intercepted, corrupted, lost, destroyed, and
incomplete and / or arrive late. It could contain viruses when transmitted
over the public networks. If you suspect that the e-mail may have been
intercepted or amended, please contact the sender. This message has been
scanned for viruses. You should virus-check attachments before opening.
When addressed to i for change ltd.’s clients, any opinion or advice
contained in this e-mail is subject to i for change ltd.’s terms of
business. This e-mail is provided for informational purposes only and
should not be construed as a solicitation or offer to buy or sell
securities or related financial instruments.

Registered Office: i for change ltd., 19 Rookfield Avenue, London, N10 3TS

looking at the METADATA file in your wheel distributed-1.15.0rc1-py2.py3-none-any.whl:
 ````Metadata-Version: 2.0
Name: distributed
Version: 1.15.0rc1
Summary: Distributed computing
Home-page: https://distributed.readthedocs.io/en/latest/
Author: Matthew Rocklin
Author-email: mrocklin@gmail.com
License: BSD
Platform: UNKNOWN
Requires-Dist: click (>=6.6)
Requires-Dist: cloudpickle (>=0.2.1)
Requires-Dist: dask (>=0.12.0)
Requires-Dist: futures
Requires-Dist: locket
Requires-Dist: msgpack-python
Requires-Dist: psutil
Requires-Dist: six
Requires-Dist: sortedcollections
Requires-Dist: tblib
Requires-Dist: toolz (>=0.7.4)
Requires-Dist: tornado (>=4.2)
Requires-Dist: zict (>=0.1.1)

UNKNOWN
````

there is no **python_version < '3.0'** in it
Maybe the "futures" added by code  in your setup.py is overriding the one in the requirements.txt.

Supposing you're right, you should apply also the trick to "locket" and remove the code 
````
if sys.version_info[0] < 3:
    requires.append('futures')
    requires.append('locket')
````

then see if the METADATA (or metadata.json) of your next universal wheel is keeping this conditional requirement
See https://github.com/dask/distributed/pull/772
That is a great simplification for setup.py writing. Did the final Wheel keep the "python_version"  (PEP 508 style) test ?

I haven't yet built a wheel.  I also don't know how to check for python_version tests.
(My knowledge of packaging is probably less than what it ought to be)
if it's like PEP 426, we should see it in METADATA and/or metadata.json.

example in (unzipped) wheel-0.29.0-py2.py3-none-any.whl METADATA : 
````
Metadata-Version: 2.0
Name: wheel
Version: 0.29.0
Summary: A built-package format for Python.
Home-page: https://bitbucket.org/pypa/wheel/
Author: Daniel Holth
Author-email: dholth@fastmail.fm
License: MIT
Keywords: wheel,packaging
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.2
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Requires-Dist: argparse; python_version=="2.6"
Provides-Extra: faster-signatures
Requires-Dist: ed25519ll; extra == 'faster-signatures'
Provides-Extra: signatures
Requires-Dist: keyring; extra == 'signatures'
Requires-Dist: keyrings.alt; extra == 'signatures'
Provides-Extra: signatures
Requires-Dist: importlib; python_version=="2.6" and extra == 'signatures'
Provides-Extra: signatures
Requires-Dist: pyxdg; sys_platform!="win32" and extra == 'signatures'
Provides-Extra: tool
````


Factual test will be to pip install the universal wheel from a python-3.6 and see if "futures" is claimed again.


```
Metadata-Version: 2.0
Name: distributed
Version: 1.14.3+117.g3480370
Summary: Distributed computing
Home-page: https://distributed.readthedocs.io/en/latest/
Author: Matthew Rocklin
Author-email: mrocklin@gmail.com
License: BSD
Platform: UNKNOWN
Requires-Dist: click (>=6.6)
Requires-Dist: cloudpickle (>=0.2.1)
Requires-Dist: dask (>=0.12.0)
Requires-Dist: futures
Requires-Dist: msgpack-python
Requires-Dist: psutil
Requires-Dist: singledispatch
Requires-Dist: six
Requires-Dist: sortedcollections
Requires-Dist: tblib
Requires-Dist: toolz (>=0.7.4)
Requires-Dist: tornado (>=4.2)
Requires-Dist: zict (>=0.1.1)

UNKNOWN
```

It appears that the wheel does not include the test.
Perhaps the `--universal` flag is no longer valid
This morning, I check with my own library:
- in the setup.py 
`install_requires=["singledispatch; python_version < '3.4'"],`
- building from python 3.6:
`python setup.py sdist bdist_wheel  --universal`
- result in METADATA:
`Requires-Dist: singledispatch`

==> The wheel format looses the PEP508 conditional information on python_version

So doing a py2 wheel and  a py3 wheel would mitigate the problem, as the wheels will work for Python-2.7/3.4/3.5/3.6

PEP 508 remembers that wheel format is limited to PEP 426 specification.

I don't find a change in wheel format 1.9 PEP 491, so it looks like a long term wheel 2.0 format thing
hum, there seem to be no filter in metadata, whatever the python version that generates the wheel.

There will be the need to remove by code from "install_requires" the not-python-3 packages when creating a py3 wheel
Another option here is to just not care.  Installing futures in Python 3 seems to be mostly innocuous.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -6,6 +6,18 @@
 import versioneer
 
 requires = open('requirements.txt').read().strip().split('\n')
+install_requires = []
+extras_require = {}
+for r in requires:
+    if ';' in r:
+        # requirements.txt conditional dependencies need to be reformatted for wheels
+        # to the form: `'[extra_name]:condition' : ['requirements']`
+        req, cond = r.split(';', 1)
+        cond = ':' + cond
+        cond_reqs = extras_require.setdefault(cond, [])
+        cond_reqs.append(req)
+    else:
+        install_requires.append(r)
 
 setup(name='distributed',
       version=versioneer.get_version(),
@@ -17,7 +29,8 @@
       license='BSD',
       package_data={ '': ['templates/index.html'], },
       include_package_data=True,
-      install_requires=requires,
+      install_requires=install_requires,
+      extras_require=extras_require,
       packages=['distributed',
                 'distributed.bokeh',
                 'distributed.bokeh.background',


Test patch: