Repository: PrefectHQ/prefect

Issue:
LocalDaskExecutor(scheduler="threads") does not fail when task has timeout
## Description
<!-- A clear description of the bug -->
`LocalDaskExecutor(scheduler="threads")` does not fail when task has timeout.

## Reproduction
<!-- A minimal example that exhibits the behavior. -->
```python
import os

os.environ["PREFECT__LOGGING__LEVEL"] = "DEBUG"

from prefect import task, Flow, executors
from prefect.engine import signals


@task(timeout=10)
def task_test():
    raise signals.FAIL("it failed")


with Flow("test_processes", executor=executors.LocalDaskExecutor(scheduler="processes")) as flow:
    _ = task_test(task_args={"name": "task_processes"})
assert flow.run().is_failed()

# NOTE: this flow should not be successful
with Flow("test_threads", executor=executors.LocalDaskExecutor(scheduler="threads")) as flow:
    _ = task_test(task_args={"name": "task_threads"})
assert flow.run().is_successful()
```

<details>
<summary>logs:</summary>

```bash
INFO - prefect.FlowRunner | Beginning Flow run for 'test_processes'
DEBUG - prefect.FlowRunner | Using executor type LocalDaskExecutor
DEBUG - prefect.FlowRunner | Flow 'test_processes': Handling state change from Scheduled to Running
INFO - prefect.TaskRunner | Task 'task_processes': Starting task run...
DEBUG - prefect.TaskRunner | Task 'task_processes': Handling state change from Pending to Running
DEBUG - prefect.TaskRunner | Task 'task_processes': Calling task.run() method...
DEBUG - prefect.TaskRunner | Task 'task_processes': Attaching thread based timeout handler...
DEBUG - prefect.TaskRunner | Task 'task_processes': Sending alarm with 10s timeout...
DEBUG - prefect.TaskRunner | Task 'task_processes': Executing function in main thread...
INFO - prefect.TaskRunner | FAIL signal raised: FAIL('it failed')
DEBUG - prefect.TaskRunner | Task 'task_processes': Handling state change from Running to Failed
INFO - prefect.TaskRunner | Task 'task_processes': Finished task run for task with final state: 'Failed'
INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
DEBUG - prefect.FlowRunner | Flow 'test_processes': Handling state change from Running to Failed

INFO - prefect.FlowRunner | Beginning Flow run for 'test_threads'
DEBUG - prefect.FlowRunner | Using executor type LocalDaskExecutor
DEBUG - prefect.FlowRunner | Flow 'test_threads': Handling state change from Scheduled to Running
INFO - prefect.TaskRunner | Task 'task_threads': Starting task run...
DEBUG - prefect.TaskRunner | Task 'task_threads': Handling state change from Pending to Running
DEBUG - prefect.TaskRunner | Task 'task_threads': Calling task.run() method...
DEBUG - prefect.TaskRunner | Task 'task_threads': Attaching process based timeout handler...
DEBUG - prefect.TaskRunner | Task 'task_threads': Sending execution to a new process...
DEBUG - prefect.TaskRunner | Task 'task_threads': Waiting for process to return with 10s timeout...
DEBUG - prefect.TaskRunner | Task 'task_threads': Executing...
ERROR - prefect.TaskRunner | Task 'task_threads': Encountered a PrefectSignal FAIL, returning details as a result...
DEBUG - prefect.TaskRunner | Task 'task_threads': Pickling value of size 88...
DEBUG - prefect.TaskRunner | Task 'task_threads': Pickling successful!
DEBUG - prefect.TaskRunner | Task 'task_threads': Passing result back to main process...
DEBUG - prefect.TaskRunner | Task 'task_threads': Result received from subprocess, unpickling...
DEBUG - prefect.TaskRunner | Task 'task_threads': Handling state change from Running to Success
INFO - prefect.TaskRunner | Task 'task_threads': Finished task run for task with final state: 'Success'
INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
DEBUG - prefect.FlowRunner | Flow 'test_threads': Handling state change from Running to Success
```

</details>



## Environment
prefect: 1.2.1


Discussion:
@mgsnuno What operating system are you using? Can you include debug level logs? They will include information about the timeout handler.
env: linux | vscode | jupyter interactive window

```bash
[2022-05-17 15:46:53+0000] INFO - prefect.FlowRunner | Beginning Flow run for 'test'
[2022-05-17 15:46:53+0000] INFO - prefect.TaskRunner | Task 'task_test': Starting task run...
[2022-05-17 15:46:53+0000] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('it failed')
[2022-05-17 15:46:53+0000] INFO - prefect.TaskRunner | Task 'task_test': Finished task run for task with final state: 'Failed'
[2022-05-17 15:46:53+0000] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
[2022-05-17 15:46:53+0000] INFO - prefect.FlowRunner | Beginning Flow run for 'test'
[2022-05-17 15:46:53+0000] INFO - prefect.TaskRunner | Task 'task_test': Starting task run...
[2022-05-17 15:46:53+0000] DEBUG - prefect.TaskRunner | Task 'task_test': Executing...
[2022-05-17 15:46:53+0000] ERROR - prefect.TaskRunner | Task 'task_test': Encountered a PrefectSignal FAIL, returning details as a result...
[2022-05-17 15:46:53+0000] DEBUG - prefect.TaskRunner | Task 'task_test': Pickling value of size 88...
[2022-05-17 15:46:53+0000] DEBUG - prefect.TaskRunner | Task 'task_test': Pickling successful!
[2022-05-17 15:46:53+0000] DEBUG - prefect.TaskRunner | Task 'task_test': Passing result back to main process...
[2022-05-17 15:46:54+0000] INFO - prefect.TaskRunner | Task 'task_test': Finished task run for task with final state: 'Success'
[2022-05-17 15:46:54+0000] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
```

your question triggered me to try it just running a script with `python test.py` and the behavior is different.


So here the issue is that raising a signal is not respected when using a timeout on a threaded worker, not that the timeout is not respected, yes?
indeed, it became apparent to me when tasks were being marked as successful even though a FAIL signal was being raised, well within the established timeout. And this seems tied only to a threaded worker when a timeout is specified.


Another question would be: which executor to use in order to benefit from an actual timeout, not a soft_timeout?

[EDIT] - updated the code and with proper debug logs. it looks like when we run with `scheduler="processes"` the tasks are `Attaching thread based timeout handler...`, and the other way around for `scheduler="threads"`

Resolving patch:
diff --git a/src/prefect/utilities/executors.py b/src/prefect/utilities/executors.py
--- a/src/prefect/utilities/executors.py
+++ b/src/prefect/utilities/executors.py
@@ -413,7 +413,7 @@ def run_with_multiprocess_timeout(
         pickled_result = queue.get(block=True, timeout=timeout)
         logger.debug(f"{name}: Result received from subprocess, unpickling...")
         result = cloudpickle.loads(pickled_result)
-        if isinstance(result, Exception):
+        if isinstance(result, (Exception, PrefectSignal)):
             raise result
         return result
     except Empty:


Test patch:
diff --git a/tests/utilities/test_executors.py b/tests/utilities/test_executors.py
--- a/tests/utilities/test_executors.py
+++ b/tests/utilities/test_executors.py
@@ -17,6 +17,7 @@
     RecursiveCall,
     HeartbeatThread,
 )
+from prefect.engine.signals import FAIL
 
 # We will test the low-level timeout handlers here and `run_task_with_timeout`
 # is covered in `tests.core.test_flow.test_timeout_actually_stops_execution`
@@ -110,7 +111,7 @@ def do_nothing(x, y=None):
     sys.platform == "win32", reason="Windows doesn't support any timeout logic"
 )
 @pytest.mark.parametrize("timeout_handler", TIMEOUT_HANDLERS)
-def test_timeout_handler_reraises(timeout_handler):
+def test_timeout_handler_reraises_exceptions(timeout_handler):
     def do_something():
         raise ValueError("test")
 
@@ -118,6 +119,18 @@ def do_something():
         timeout_handler(do_something, timeout=10)
 
 
+@pytest.mark.skipif(
+    sys.platform == "win32", reason="Windows doesn't support any timeout logic"
+)
+@pytest.mark.parametrize("timeout_handler", TIMEOUT_HANDLERS)
+def test_timeout_handler_reraises_prefect_signals(timeout_handler):
+    def do_something():
+        raise FAIL("test")
+
+    with pytest.raises(FAIL, match="test"):
+        timeout_handler(do_something, timeout=10)
+
+
 # Define a top-level helper function for a null-op process target, must be defined
 # as a non-local for the python native pickler used within `my_process`
 def do_nothing():