Repository: apache/airflow

Issue:
Allow to set execution_timeout default value in airflow.cfg
### Discussed in https://github.com/apache/airflow/discussions/18411

<div type='discussions-op-text'>

<sup>Originally posted by **alexInhert** September 21, 2021</sup>
### Description

Currently the default value of execution_timeout in base operator is None
https://github.com/apache/airflow/blob/c686241f4ceb62d52e9bfa607822e4b7a3c76222/airflow/models/baseoperator.py#L502

This means that a task will run without limit till finished.


### Use case/motivation

The problem is that there is no way to overwrite this default for all dags. This causes problems where we find that tasks run sometimes for 1 week!!! for no reason. They are just stuck. This mostly happens with tasks that submit work to some 3rd party resource. The 3rd party had some error and terminated but the Airflow task for some reason did not.

This can be handled in cluster policy however it feels that cluster policy is more to enforce something but less about setting defaults values.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
</div>


Discussion:
cc: @alexinhert
Details in #18411 
Hello, @potiuk I would like to work on this.
Could you guide me more on this?
> Hello, @potiuk I would like to work on this. Could you guide me more on this?

Just read the discusion in #18411 i think @alexInhert explained it in the last comment -and maybe he can provide more guidance if needed?
@eladkal as discussed #18610 I'd like to take this one on
@zevaverbach are you still working on this issue?
Oh darn, to be honest I think I over-committed. Can you please unassign me?

On Thu, Nov 18, 2021 at 6:53 PM eladkal ***@***.***> wrote:

> @zevaverbach <https://github.com/zevaverbach> are you still working on
> this issue?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/18578#issuecomment-973114986>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAVVCNBEMBTLWBXUWAKPPOTUMU4RDANCNFSM5E5Z4S5A>
> .
>

I'd be glad to take this one over.
@eladkal 
> I'd be glad to take this one over. @eladkal

Assigned
@dillonjohnson are you still working on this issue?
Yes. Sorry.
I just need to test a little bit more.
Will try to wrap it up soon. @eladkal 
On second thought, have had some stuff come up and I'm having tons of issues with the various tests failing (not sure why).
I think I'll have to pass it off. Sorry about that. @eladkal 

Resolving patch:
diff --git a/airflow/configuration.py b/airflow/configuration.py
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -15,6 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
+import datetime
 import functools
 import json
 import logging
@@ -608,6 +609,40 @@ def getjson(self, section, key, fallback=_UNSET, **kwargs) -> Union[dict, list,
         except JSONDecodeError as e:
             raise AirflowConfigException(f'Unable to parse [{section}] {key!r} as valid json') from e
 
+    def gettimedelta(self, section, key, fallback=None, **kwargs) -> Optional[datetime.timedelta]:
+        """
+        Gets the config value for the given section and key, and converts it into datetime.timedelta object.
+        If the key is missing, then it is considered as `None`.
+
+        :param section: the section from the config
+        :param key: the key defined in the given section
+        :param fallback: fallback value when no config value is given, defaults to None
+        :raises AirflowConfigException: raised because ValueError or OverflowError
+        :return: datetime.timedelta(seconds=<config_value>) or None
+        """
+        val = self.get(section, key, fallback=fallback, **kwargs)
+
+        if val:
+            # the given value must be convertible to integer
+            try:
+                int_val = int(val)
+            except ValueError:
+                raise AirflowConfigException(
+                    f'Failed to convert value to int. Please check "{key}" key in "{section}" section. '
+                    f'Current value: "{val}".'
+                )
+
+            try:
+                return datetime.timedelta(seconds=int_val)
+            except OverflowError as err:
+                raise AirflowConfigException(
+                    f'Failed to convert value to timedelta in `seconds`. '
+                    f'{err}. '
+                    f'Please check "{key}" key in "{section}" section. Current value: "{val}".'
+                )
+
+        return fallback
+
     def read(self, filenames, encoding=None):
         super().read(filenames=filenames, encoding=encoding)
 
diff --git a/airflow/models/abstractoperator.py b/airflow/models/abstractoperator.py
--- a/airflow/models/abstractoperator.py
+++ b/airflow/models/abstractoperator.py
@@ -67,6 +67,9 @@
     conf.get("core", "default_task_weight_rule", fallback=WeightRule.DOWNSTREAM)
 )
 DEFAULT_TRIGGER_RULE: TriggerRule = TriggerRule.ALL_SUCCESS
+DEFAULT_TASK_EXECUTION_TIMEOUT: datetime.timedelta = conf.gettimedelta(
+    "core", "default_task_execution_timeout"
+)
 
 
 class AbstractOperator(LoggingMixin, DAGNode):
diff --git a/airflow/models/baseoperator.py b/airflow/models/baseoperator.py
--- a/airflow/models/baseoperator.py
+++ b/airflow/models/baseoperator.py
@@ -65,6 +65,7 @@
     DEFAULT_QUEUE,
     DEFAULT_RETRIES,
     DEFAULT_RETRY_DELAY,
+    DEFAULT_TASK_EXECUTION_TIMEOUT,
     DEFAULT_TRIGGER_RULE,
     DEFAULT_WEIGHT_RULE,
     AbstractOperator,
@@ -201,7 +202,7 @@ def partial(
     queue: str = DEFAULT_QUEUE,
     pool: Optional[str] = None,
     pool_slots: int = DEFAULT_POOL_SLOTS,
-    execution_timeout: Optional[timedelta] = None,
+    execution_timeout: Optional[timedelta] = DEFAULT_TASK_EXECUTION_TIMEOUT,
     retry_delay: Union[timedelta, float] = DEFAULT_RETRY_DELAY,
     retry_exponential_backoff: bool = False,
     priority_weight: int = DEFAULT_PRIORITY_WEIGHT,
@@ -706,7 +707,7 @@ def __init__(
         pool: Optional[str] = None,
         pool_slots: int = DEFAULT_POOL_SLOTS,
         sla: Optional[timedelta] = None,
-        execution_timeout: Optional[timedelta] = None,
+        execution_timeout: Optional[timedelta] = DEFAULT_TASK_EXECUTION_TIMEOUT,
         on_execute_callback: Optional[TaskStateChangeCallback] = None,
         on_failure_callback: Optional[TaskStateChangeCallback] = None,
         on_success_callback: Optional[TaskStateChangeCallback] = None,


Test patch:
diff --git a/tests/core/test_configuration.py b/tests/core/test_configuration.py
--- a/tests/core/test_configuration.py
+++ b/tests/core/test_configuration.py
@@ -16,6 +16,7 @@
 # specific language governing permissions and limitations
 # under the License.
 import copy
+import datetime
 import io
 import os
 import re
@@ -837,3 +838,67 @@ def test_as_dict_respects_sensitive_cmds(self):
         else:
             assert 'sql_alchemy_conn' in conf_maintain_cmds['core']
             assert conf_maintain_cmds['core']['sql_alchemy_conn'] == conf_conn
+
+    def test_gettimedelta(self):
+        test_config = '''
+[invalid]
+# non-integer value
+key1 = str
+
+# fractional value
+key2 = 300.99
+
+# too large value for C int
+key3 = 999999999999999
+
+[valid]
+# negative value
+key4 = -1
+
+# zero
+key5 = 0
+
+# positive value
+key6 = 300
+
+[default]
+# Equals to None
+key7 =
+'''
+        test_conf = AirflowConfigParser(default_config=test_config)
+        with pytest.raises(
+            AirflowConfigException,
+            match=re.escape(
+                'Failed to convert value to int. Please check "key1" key in "invalid" section. '
+                'Current value: "str".'
+            ),
+        ):
+            test_conf.gettimedelta("invalid", "key1")
+
+        with pytest.raises(
+            AirflowConfigException,
+            match=re.escape(
+                'Failed to convert value to int. Please check "key2" key in "invalid" section. '
+                'Current value: "300.99".'
+            ),
+        ):
+            test_conf.gettimedelta("invalid", "key2")
+
+        with pytest.raises(
+            AirflowConfigException,
+            match=re.escape(
+                'Failed to convert value to timedelta in `seconds`. '
+                'Python int too large to convert to C int. '
+                'Please check "key3" key in "invalid" section. Current value: "999999999999999".'
+            ),
+        ):
+            test_conf.gettimedelta("invalid", "key3")
+
+        assert isinstance(test_conf.gettimedelta('valid', 'key4'), datetime.timedelta)
+        assert test_conf.gettimedelta('valid', 'key4') == datetime.timedelta(seconds=-1)
+        assert isinstance(test_conf.gettimedelta('valid', 'key5'), datetime.timedelta)
+        assert test_conf.gettimedelta('valid', 'key5') == datetime.timedelta(seconds=0)
+        assert isinstance(test_conf.gettimedelta('valid', 'key6'), datetime.timedelta)
+        assert test_conf.gettimedelta('valid', 'key6') == datetime.timedelta(seconds=300)
+        assert isinstance(test_conf.gettimedelta('default', 'key7'), type(None))
+        assert test_conf.gettimedelta('default', 'key7') is None