Repository: DataBiosphere/toil

Issue:
Toil fails on python 3.7 due to dependency
Toil has an indirect dependency on  http_parser package which is no longer maintained on pypi (last update 2013).  This package does not compile with python 3.7.  It is required by pymesos

However, it does not appear that pymesos has released a version that corrects this.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-489)
┆Issue Number: TOIL-489



Discussion:
ticket filed on pymesos:
https://github.com/douban/pymesos/issues/122
I think `pymesos` is only pulled in by the mesos extra. A workaround might be to install without the mesos extra (i.e. not `toil[all]` or `toil[mesos]`).

Sadly this won't work when building the appliance; it kind of has to support mesos.

I am changing Cactus not include mesos for now, although if you do:

    CPPFLAGS='-DPYPY_VERSION' pip install http_parser

it actually compiles (not sure if it works).  also,

    pip install git+https://github.com/benoitc/http-parser.git

work too.

sigh

I can't replicate pip install git+https://github.com/benoitc/http-parser.git working on 3.7. It still fails with a bunch of errors like:

```
    http_parser/parser.c:7038:13: error: no member named 'exc_traceback' in 'struct _ts'; did you mean 'curexc_traceback'?
        tstate->exc_traceback = local_tb;
                ^~~~~~~~~~~~~
                curexc_traceback
    /opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:241:15: note: 'curexc_traceback' declared here
        PyObject *curexc_traceback;
                  ^
```

I'll see if any of the open PRs fix it.

Odd it didn't work for you.  I tried that because it was recommend by someone in cyberspace

Using 

  CPPFLAGS='-DPYPY_VERSION'

does work, as it avoids the problem code path.

Really, pymesos shouldn't be using a DOA package ;-(

Adam Novak <notifications@github.com> writes:
> I can't replicate pip install git+https://github.com/benoitc/http-parser.git working on 3.7. It still fails with a bunch of errors like:
> 
> ```
>     http_parser/parser.c:7038:13: error: no member named 'exc_traceback' in 'struct _ts'; did you mean 'curexc_traceback'?
>         tstate->exc_traceback = local_tb;
>                 ^~~~~~~~~~~~~
>                 curexc_traceback
>     /opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:241:15: note: 'curexc_traceback' declared here
>         PyObject *curexc_traceback;
>                   ^
> ```
> 
> I'll see if any of the open PRs fix it.
> 
> -- 
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub:
> https://github.com/DataBiosphere/toil/issues/2924#issuecomment-577314589
> I can't replicate pip install git+https://github.com/benoitc/http-parser.git
> working on 3.7. It still fails with a bunch of errors like:
> 
>     http_parser/parser.c:7038:13: error: no member named 'exc_traceback' in 'struct _ts'; did you mean 'curexc_traceback'?
>         tstate->exc_traceback = local_tb;
>                 ^~~~~~~~~~~~~
>                 curexc_traceback
>     /opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:241:15: note: 'curexc_traceback' declared here
>         PyObject *curexc_traceback;
>                   ^
> 
> I'll see if any of the open PRs fix it.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.*
> 

The reason it didn't work was that I had an old version of Cython installed, which the http-parser build process decided to use to produce broken versions of its bindings.

I have a PR open to http-parser to set it up to require a compatible Cython when installing from a non-release version, and I think I can pull that in as part of Toil's Mesos extra and pymesos will just use it.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -25,6 +25,11 @@ def runSetup():
     futures = 'futures==3.1.1'
     pycryptodome = 'pycryptodome==3.5.1'
     pymesos = 'pymesos==0.3.7'
+    # We need this specific http-parser that still claims to be version 0.8.3
+    # but which builds on Python 3.7+, to satisfy pymesos
+    http_parser = 'http-parser' + \
+                  '@https://github.com/adamnovak/http-parser/archive/190a17839ba229c635b59d960579451a81fe73f3.zip' + \
+                  '#sha256=3d30c84a426627e468657c44de199daee9d3210a48e392d4ad2e7497c5010949'
     psutil = 'psutil >= 3.0.1, <6'
     pynacl = 'pynacl==1.1.2'
     gcs = 'google-cloud-storage==1.6.0'
@@ -80,6 +85,7 @@ def runSetup():
     kubernetes_reqs = [
         kubernetes]
     mesos_reqs = [
+        http_parser,
         pymesos,
         psutil]
     wdl_reqs = []


Test patch: