Repository: Parsl/parsl

Issue:
Implement time limits for python apps
Requested by @lgray.


Discussion:
I've got a quick and dirty solution for this here: https://gist.github.com/yadudoc/d9dedc4ea447b1cc1cc82846cc98fd40

Essentially this would give you a `@timeout` decorator, that you can use like this : 
```python
@python_app
@timeout
def sleeper(x, timeout=None):
    """ Timeout must be passed as a kwarg at function call time
    """
    import time
    time.sleep(x)
    return x

sleeper(10, timeout=2).result()
```
As reported by @annawoodard this method can result in the worker getting killed by the interrupts, and needs more work in order to be useful.

@jchiang87 also request this feature for debugging an issue with python apps.
Using process-wide alarms rather than something restricted to the scope of the actual executing app is probably going to never work right.

There is this awful hack which lets you fire an asynchronous exception into an arbitrary thread, which could probably be used as long as you were sure of the thread executing the app. https://gist.github.com/liuw/2407154



Resolving patch:
diff --git a/parsl/app/app.py b/parsl/app/app.py
--- a/parsl/app/app.py
+++ b/parsl/app/app.py
@@ -67,6 +67,8 @@ def __init__(self, func, data_flow_kernel=None, walltime=60, executors='all', ca
             self.kwargs['stdout'] = params['stdout'].default
         if 'stderr' in params:
             self.kwargs['stderr'] = params['stderr'].default
+        if 'walltime' in params:
+            self.kwargs['walltime'] = params['walltime'].default
         self.outputs = params['outputs'].default if 'outputs' in params else []
         self.inputs = params['inputs'].default if 'inputs' in params else []
 
diff --git a/parsl/app/python.py b/parsl/app/python.py
--- a/parsl/app/python.py
+++ b/parsl/app/python.py
@@ -12,6 +12,27 @@
 logger = logging.getLogger(__name__)
 
 
+def timeout(f, seconds):
+    def wrapper(*args, **kwargs):
+        import threading
+        import ctypes
+        import parsl.app.errors
+
+        def inject_exception(thread):
+            ctypes.pythonapi.PyThreadState_SetAsyncExc(
+                ctypes.c_long(thread),
+                ctypes.py_object(parsl.app.errors.AppTimeout)
+            )
+
+        thread = threading.current_thread().ident
+        timer = threading.Timer(seconds, inject_exception, args=[thread])
+        timer.start()
+        result = f(*args, **kwargs)
+        timer.cancel()
+        return result
+    return wrapper
+
+
 class PythonApp(AppBase):
     """Extends AppBase to cover the Python App."""
 
@@ -45,6 +66,9 @@ def __call__(self, *args, **kwargs):
         else:
             dfk = self.data_flow_kernel
 
+        walltime = self.kwargs.get('walltime')
+        if walltime is not None:
+            self.func = timeout(self.func, walltime)
         app_fut = dfk.submit(self.func, *args,
                              executors=self.executors,
                              fn_hash=self.func_hash,


Test patch:
diff --git a/parsl/tests/test_error_handling/test_python_walltime.py b/parsl/tests/test_error_handling/test_python_walltime.py
new file mode 100644
--- /dev/null
+++ b/parsl/tests/test_error_handling/test_python_walltime.py
@@ -0,0 +1,20 @@
+import pytest
+
+import parsl
+from parsl.app.errors import AppTimeout
+
+
+@parsl.python_app
+def my_app(walltime=3):
+    import time
+    # the loop count must be substantially bigger than the walltime
+    # but not infinite - so that the test eventually terminates
+    # even if the walltime doesn't work.
+    for n in range(0, 6):
+        time.sleep(1)
+
+
+def test_python_walltime():
+    f = my_app()
+    with pytest.raises(AppTimeout):
+        f.result()