Repository: Parsl/parsl

Issue:
using walltime for python app results in Recursion Error
I've been trying to use the timeout functionality of parsl via the walltime=X argument. To narrow it down to a minimum example, I took the test in the test suite and modified it to invoke the app many times:

```
import parsl
from parsl.app.errors import AppTimeout
import pytest
from parsl.configs.local_threads import config

from parsl.providers import LocalProvider
from parsl.channels import LocalChannel
from parsl.config import Config
from parsl.executors import HighThroughputExecutor


local_htex = Config(
    executors=[
        HighThroughputExecutor(
            label="htex_Local",
            worker_debug=True,
            cores_per_worker=1,
            provider=LocalProvider(
                channel=LocalChannel(),
                init_blocks=1,
                max_blocks=1,
            ),
        )
    ],
    strategy=None,
)

parsl.load(local_htex)


@parsl.python_app
def my_app(duration, bar=1.0, walltime=1):
    import time
    time.sleep(duration*bar)
    return True


def test_python_walltime():
    fs = []
    for i in range(1024):
        f = my_app(2.0)
        fs.append(f)
    print("all invoked")
    for f in fs:
        with pytest.raises(AppTimeout):
            f.result()

if __name__ == "__main__":
    test_python_walltime()

```

This gives a (deep) stack trace with repeated 
```
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 393, in can
    return canner(obj)
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 186, in __init__
    self.closure = tuple( can(cell) for cell in closure )
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 186, in <genexpr>
    self.closure = tuple( can(cell) for cell in closure )
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 393, in can
    return canner(obj)
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 165, in __init__
    self.cell_contents = can(cell.cell_contents)
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 393, in can
    return canner(obj)
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 186, in __init__
    self.closure = tuple( can(cell) for cell in closure )
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 186, in <genexpr>
    self.closure = tuple( can(cell) for cell in closure )
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 393, in can
    return canner(obj)
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 165, in __init__
    self.cell_contents = can(cell.cell_contents)
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 393, in can
    return canner(obj)
  File "/work/04372/ejonas/stampede2/anaconda/envs/nmr-abinitio/lib/python3.7/site-packages/ipyparallel/serialize/canning.py", line 177, in __init__
    self._check_type(f)
RecursionError: maximum recursion depth exceeded

```

style of errors. Note that this only appears when the `walltime` arg is present in `my_app` -- if you delete that, this example works flawlessly.  Also note the recursion error does NOT occur when using the local thread executor. 

I currently see this with both master and the pip-installable version of parsl on a linux host (tacc login node) installed via the newest anaconda with python 3.7. 

Unfortunately this is currently blocking experiments as I have ~1% of my jobs that take 100x longer than the others and thus I'd like them to die when they exceed a limit. 



Discussion:

Resolving patch:
diff --git a/parsl/app/python.py b/parsl/app/python.py
--- a/parsl/app/python.py
+++ b/parsl/app/python.py
@@ -63,8 +63,11 @@ def __call__(self, *args, **kwargs):
 
         walltime = self.kwargs.get('walltime')
         if walltime is not None:
-            self.func = timeout(self.func, walltime)
-        app_fut = dfk.submit(self.func, *args,
+            func = timeout(self.func, walltime)
+        else:
+            func = self.func
+
+        app_fut = dfk.submit(func, *args,
                              executors=self.executors,
                              fn_hash=self.func_hash,
                              cache=self.cache,


Test patch: